Testing

Testing of the whole ELK Stack can be easily done using Vagrant.

Note

Recall that when having a relatively large amount of logs, Elasticsearch and Logstash use about 4-6GB RAM (or even more) when filtering and indexing data.

The vagrantfile looks like this:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|

  config.vm.define "cr0n05", autostart: false do |cronos|

    cronos.vm.box = "centos/7"
    cronos.vm.network "private_network", ip: "192.168.1.2"
    cronos.vm.hostname = "cr0n05"

    cronos.vm.provider "virtualbox" do |v|
      v.memory = 1024
    end

    cronos.vm.provision "ansible_local" do |ansible|
      ansible.become = true
      ansible.playbook = "site.yml"
      ansible.verbose = "vv"
      ansible.extra_vars = {
        machine: "cr0n05",
        elk_ip: "192.168.1.4",
        elk_hostname: "elk"
      }
    end

  end

  config.vm.define "4p0l0", autostart: false do |apolo|

    apolo.vm.box = "centos/7"
    apolo.vm.network "private_network", ip: "192.168.1.3"
    apolo.vm.hostname = "4p0l0"

    apolo.vm.provider "virtualbox" do |v|
      v.memory = 1024
    end

    apolo.vm.provision "ansible_local" do |ansible|
      ansible.become = true
      ansible.playbook = "site.yml"
      ansible.verbose = "vv"
      ansible.extra_vars = {
        machine: "4p0l0",
        elk_ip: "192.168.1.4",
        elk_hostname: "elk"
      }
    end

  end

  config.vm.define "elk", autostart: false do |elk|

    elk.vm.box = "centos/7"
    elk.vm.network "private_network", ip: "192.168.1.4"
    elk.vm.hostname = "elk"

    elk.vm.provider "virtualbox" do |v|
      v.memory = 4096
    end

    elk.vm.provision "ansible_local" do |ansible|
      ansible.become = true
      ansible.playbook = "site.yml"
      ansible.verbose = "vv"
      ansible.extra_vars = {
        machine: "elk"
      }
    end

  end

end

For the purpose of this guide use the Ansible project here. This project is explained in Installation and Configuration. Although, in case of setting up another configuration, read the explanation below of the Vagrantfile above so that it can be replicated.

In the configuration of each virtual machine, there is a subsection for provisioning. In that subsection, there is a variable that is accessed as ansible.playbook. Set it to the path of the main ansible playbook.

Take a look at the provisioning subsection in the vagrantfile, note that the ansible.extra_vars defines a variable called machine, this variable must match the hostname of the virtual machine.

The hostname of the virtual machine can be changed with the variable vm.hostname. For more information, read the Vagrant documentation about vagrantfiles.

The variables elk_ip and elk_hostname under the configuration of 4p0l0 and cr0n05, are used to make elk visible by its hostname automatically.

The site.yml uses one playbook or another depending on the value of the variable machine:

# site.yml
---
- import_playbook: "playbooks/{{ machine }}.yml"

The playbooks/4p0l0.yml, playbooks/cr0n05.yml, and playbooks/elk.yml playbooks are simple too:

# playbooks/4p0l0.yml
---
- hosts: 4p0l0
  roles:
    - ../roles/master

# playbooks/cr0n05.yml
---
- hosts: cr0n05
  roles:
    - ../roles/master

# playbooks/elk.yml
---
- hosts: elk
  roles:
    - ../roles/elk

The roles elk and master are responsible for setting up ELK and Filebeat respectively. Go to Installation and Configuration, for a more detailed explanation.

Before starting the virtual cluster please see the directory structure that should be matched in order to run the tests here.

To start up the virtual cluster use the following bash script:

#!/bin/bash

if [ "$1" == "up" ]; then
    vagrant up elk cr0n05 4p0l0 --no-provision
elif [ "$1" == "provision-elk" ]; then
    vagrant provision elk
elif [ "$1" == "provision-filebeat" ]; then
    vagrant provision cr0n05 4p0l0
elif [ "$1" == "halt" ]; then
    vagrant halt elk cr0n05 4p0l0
else
    echo "Usage: ./run.sh up|provision-elk|provision-filebeat"
fi

From the root of the project run:

$ ./scripts/run.sh up

Now provision elk, run:
```
$ ./scripts/run.sh provision-elk
```
After correctly provisioning elk, set up the Indexes and Mappings in Kibana.

Warning

Before provisioning filebeat it is very important to set up the indexes and mappings in Kibana.
After setting up Kibana run:
```
$ ./scripts/run.sh provision-filebeat
```
If everything is ok, new logging sources can be addded, as well as, create visualizations and dashboards, etc.
To stop the cluster run:
```
$ ./scripts/run.sh halt
```