ELK Stack 6.x Installation and Configuration

Note

Installation

The architecture in which the ELK Stack was installed is the following.

    ELK Server
   ----------------
     Kibana
     Elasticsearch
     Logstash
   ----------------
          ||
          ||
          ||
  ------------------
  |                |
------------      ------------
  Filebeat          Filebeat
------------      ------------
 Beat Server       Beat Server

Also, it is important to note that the stack of applications was installed on CentOS 7 using Ansible. Therefore, in the next subsections, there will be an explanation of the tasks used to install each of the ELK Stack components.

Before proceeding to the installation of each main component, it is needed to add the ELK’s repository to the rpm’s repositories.

---
- name: Download public signing key
  shell: rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

- name: Add elk repository file
  template:
    src: "etc/yum.repos.d/elk.repo.j2"
    dest: "/etc/yum.repos.d/elk.repo"
    owner: root
    group: root
    mode: 0644

This playbook basically adds the ELK’s gpg signing key and takes a template to render it in the /etc/yum/repos.d/ directory, which is where rpm looks for its repositories. The template file is this:

[elastic-{{ elk_version }}]
name=Elastic repository for {{ elk_version }} packages
baseurl=https://artifacts.elastic.co/packages/{{ elk_version }}/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

The {{ elk_version }} jinja variable refers to the version of your desired stack. In this case 6.c. This variable must be passed as an argument when running ansible or have it defined somewhere in your ansible project. For more information about variables go to the Ansible’s documentation.

Also, it is needed to install Java 8 and the main components (elasticsearch, logstash, kibana) packages.

---
- name: Install Java
  yum:
    name: java-{{ java_version }}-openjdk-devel
    state: present
    
- name: Install logstash-elasticsearch-kibana
  yum:
    name:
      - logstash
      - elasticsearch
      - kibana
    state: present

The variable {{ java_version }} represents the java version used, in our case (and due to compatibility) 1.8.0.

Elasticsearch

After installing the needed package, Elasticsearch is configured like this:

---
- name: Configure elasticsearch
  template:
    src: "etc/elasticsearch/elasticsearch.yml.j2"
    dest: "/etc/elasticsearch/elasticsearch.yml"
    owner: root
    group: root
    mode: 0644
  notify: enable_restart_elasticsearch

The Elasticsearch main configuration file, which is a template, is rendered in /etc/elasticsearch/. The template can be found here. In that template, you will find a variable called {{ machine }}, which is rendered as the hostname of our ELK server, in our case elk. So in your case, you can use whatever you want, but from now on in this guide, we will use the hostname elk. Also, when the configuration file is placed, a notify is made so that the Elasticsearch service is started/restarted. The handler looks like this:

---
- name: enable_restart_elasticsearch
  systemd:
    name: elasticsearch
    state: restarted
    enabled: yes

Logstash

After installing the needed package, Logstash is configured like this:

---
- name: Configure logstash
  copy:
    src: "etc/logstash/{{ item }}"
    dest: "/etc/logstash/{{ item }}"
    mode: 0644
  with_items:
    - pipelines.yml
    - logstash.yml
  notify: enable_restart_logstash

- name: Logstash main pipeline configuration file
  template:
    src: "etc/logstash/conf.d/main_pipeline.conf.j2"
    dest: "/etc/logstash/conf.d/main_pipeline.conf"
    owner: root
    group: root
    mode: 0644
  notify: enable_restart_logstash

The first task copies two configuration files, pipelines.yml and logstash.yml. The first file indicates to Logstash where to find our pipelines configuration files. You can find it here. The second one is the main configuration file for Logstash. You can find it here.

The second task takes a template and renders it in the pipelines directory. The template represents the description of our main pipeline, that is, inputs, filters, and outputs. You can find it here.

Note

Logstash Filters: It is important to know the version of the filter plugins that you are using so you will be able to search for the proper documentation.

Kibana

After installing the needed package, Kibana is configured like this:

---
- name: Configure kibana
  template:
    src: "etc/kibana/kibana.yml.j2"
    dest: "/etc/kibana/kibana.yml"
    owner: root
    group: root
    mode: 0644
  notify: enable_restart_kibana

The Kibana main configuration file, which is a template, is rendered in /etc/kibana/. The template can be found here. Also, when the configuration file is placed, a notify is made so that the Kibana service is started/restarted. The handler looks like this:

---
- name: enable_restart_kibana
  systemd:
    name: kibana
    state: restarted
    enabled: yes

After installing and configuring Kibana, it is time to give structure to our logs and create/import the dashboards and visualizations needed:

  1. Access the web interface through http://elk:5601. To access it using the domain name elk remember to add elk to your hosts file.
  2. Organize the information. This will help you plot all your data easily.

Note

Create the indexes, and the mappings BEFORE sending any data to Elasticsearch.

  1. Create indexes and mappings, that is, give types and formats to your data.
    • In the Dev Tools section, copy and paste the content of the index and mappings file, then select it all and click on RUN. Note that these mappings are the ones that we use, you can take them as an example and create yours, for more information go to ELK’s documentation about mappings.
    • To easily see your mappings go to: Management -> Index management -> Select your index -> Mapping.
  2. Continue with c and d steps after Filebeat is sending information to logstash. So please go to Filebeat.
    • You can check that it is already done if you can create index patterns, that is, it won’t let you create them if you don’t have any data.
  3. Create the dashboard and visualizations.
    • Go to Management, then, under the Kibana section go to Saved Objects, then, Import, and import the dashboards and visualizations file.
    • If you want to export the visualizations to a JSON format, remember to export every saved object, because some visualizations may depend on other objects and they won’t work if you don’t export them all.
  4. In the section Management -> Index Patterns select one (no matter which one) index pattern and press the start button to make it the default one.

Filebeat

Remember that in our case Filebeat is installed in servers different from the ELK Server, see Installation.

So, the installation playbook looks like this:

---
- name: Download public signing key
  shell: rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

- name: Add elk repository file
  template:
    src: "etc/yum.repos.d/elk.repo.j2"
    dest: "/etc/yum.repos.d/elk.repo"
    owner: root
    group: root
    mode: 0644
    
- name: Install filebeat
  yum:
    name: filebeat
    state: present

- name: Configure filebeat
  template:
    src: etc/filebeat/filebeat.yml.j2
    dest: /etc/filebeat/filebeat.yml
    owner: root
    group: rot
    mode: 0644
  notify: enable_restart_filebeat

As previously explained, the three first tasks are for adding the ELK’s repository and installing the main component package. The last task is for configuring Filebeat. It takes a template file, which contains the Filebeat main configuration, that is, where it will take the logs from. You can find the template file here. Then, after Filebeat is configured, a notification is sent to a handler to start/restart the Filebeat service. The handler looks like this:

---
- name: enable_restart_filebeat
  service:
    name: filebeat
    state: restarted
    enabled: yes

Testing

If you want to test all the ELK Stack locally you can easily do it using Vagrant.

Note

Elasticsearch and Logstash use together at least 5 GB of RAM when not in idle state.

The vagrantfile looks like this:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|

  config.vm.define "cr0n05", autostart: false do |cronos|
    
    cronos.vm.box = "centos/7"
    cronos.vm.network "private_network", ip: "192.168.1.2"
    cronos.vm.hostname = "cr0n05"

    cronos.vm.provider "virtualbox" do |v|
      v.memory = 1024
    end

    cronos.vm.provision "ansible_local" do |ansible|
      ansible.become = true
      ansible.playbook = "site.yml"
      ansible.verbose = "vv"
      ansible.extra_vars = {
        machine: "cr0n05"
      }
    end
    
  end

  config.vm.define "4p0l0", autostart: false do |apolo|
    
    apolo.vm.box = "centos/7"
    apolo.vm.network "private_network", ip: "192.168.1.3"
    apolo.vm.hostname = "4p0l0"

    apolo.vm.provider "virtualbox" do |v|
      v.memory = 1024
    end

    apolo.vm.provision "ansible_local" do |ansible|
      ansible.become = true
      ansible.playbook = "site.yml"
      ansible.verbose = "vv"
      ansible.extra_vars = {
        machine: "4p0l0"
      }
    end
    
  end

  config.vm.define "elk", autostart: false do |elk|
    
    elk.vm.box = "centos/7"
    elk.vm.network "private_network", ip: "192.168.1.4"
    elk.vm.hostname = "elk"

    elk.vm.provider "virtualbox" do |v|
      v.memory = 4096
    end

    elk.vm.provision "ansible_local" do |ansible|
      ansible.become = true
      ansible.playbook = "site.yml"
      ansible.verbose = "vv"
      ansible.extra_vars = {
        machine: "elk"
      }
    end
    
  end
  
end

In the configuration of each virtual machine, there is a subsection for provisioning. In that subsection, there is a variable that is accessed as ansible.playbook. You have to set it to the path to your ansible playbook. You should use the playbook that was explained in the previous section, Installation. Also in this provisioning subsection, note that the ansible.extra_vars defines a variable called machine, so if you are using the playbook explained before, this variable must match the hostname of the virtual machine. The hostname of the virtual machine can be changed with the variable vm.hostname. For more information read the Vagrant documentation about vagrantfiles.

To start up the virtual cluster use the following bash script with the argument up:

#!/bin/bash

if [ "$1" == "up" ]; then
    vagrant up elk cr0n05 4p0l0 --no-provision
elif [ "$1" == "provision-elk" ]; then
    vagrant provision elk
elif [ "$1" == "provision-filebeat" ]; then
    vagrant provision cr0n05 4p0l0    
else
    echo "Usage: ./run.sh up|provision-elk|provision-filebeat"
fi

Note

Change elk, cr0n05, 4p0l0, to the virtual machine names that you set up in your Vagrantfile. If you are using the vagrantfile from above, you do not have to change them.

Make the virtual machines visible between them by their hostname. You just have to change the /etc/hosts file and add the ip address of the virtual machine that you want to see followed by its hostname. For example, make elk visible by others and in the elk machine.

# file /etc/hosts
0.0.0.0         elk      # allow others to use the elk hostname instead of the ip
192.168.1.2     cr0n05   # make cr0n05 visible to elk by its hostname not just its ip
192.168.1.3     4p0l0

After making them visible, run the script with the argument provision-elk so that Elasticsearch, Logstash, and Kibana will be installed. Configure Kibana as explained in Kibana. Then run the script with the argument provision-filebeat. When it finishes you should be able to open your browser in the elk machine’s ip address port 5601.

Authors