1. Overview

The proposed architecture aims to provision a system by following the steps below:

1. Create ansible’s directory hierarchy, playbooks, roles, taskfiles, etc. (basically,the logic that will be used to provision the system), and maintain it under version control.

2. ssh into the master node and download the repo from version control. Consider using read-only deploy keys to download the repo without having to type an username and password; especially in unattended deployments.

3. Run a one-shot script to perform an initial setup of the environment required for ansible to run properly during and after deployment, followed by the actual software execution (see Section 3).

4. Check the bootstrap log and verify there were no errors and/or unexpected behaviors.

5. If errors arose, fix them and re-run the bootstrap script.

2. Provisioning logic

2.1. Directory hierarchy

The logic should consider constant creation of new playbooks, roles, taskfiles, etc., allowing to easy scale in the future. Take the example below:

# tree /etc/ansible
.
├── ansible.cfg
├── environments
│   ├── dev
│   │   ├── group_vars
│   │   └── inventory
│   └── prod
│       ├── group_vars
│       └── inventory
├── playbooks
│   ├── playbook_1.yml
│   ├── playbook_2.yml
│   └── playbook_n.yml
├── roles
│   ├── role_1
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   └── vars
│   │       └── main.yml
│   ├── role_2
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   └── vars
│   │       └── main.yml
│   └── role_n
│       ├── handlers
│       │   └── main.yml
│       ├── tasks
│       │   └── main.yml
│       ├── templates
│       └── vars
│           └── main.yml
├── scripts
│   ├── bootstrap.sh
│   └── vault-client.sh
└── site.yml

It is designed so that upon calling site.yml tasks from a particular set of playbooks, from the playbooks folder, are applied. This behavior can be accomplished by manually importing the playbooks or using variables:

Importing playbooks	Using variables
# /etc/ansible/site.yml --- - import_playbook: playbooks/playbook_1.yml - import_playbook: playbooks/playbook_2.yml - import_playbook: playbooks/playbook_n.yml	# /etc/ansible/environments/prod/group_vars/group1 --- playbook: playbook_1 # /etc/ansible/environments/prod/group_vars/group2 --- playbook: playbook_2 # /etc/ansible/environments/prod/group_vars/groupN --- playbook: playbook_n # /etc/ansible/site.yml --- - import_playbook: "playbooks/{{ playbook }}.yml"

One could couple all playbooks inside site.yml, but that would make future scalability difficult and potentially cause problems if a large number of people is working on the same project (take git merge conflicts for example).

If one-shot playbooks (playbooks that run only once, such as whose involving firmware updates) are to be managed, it is recommended to modify the directory hierarchy so that the playbooks folder holds them. For example:

.
└── playbooks
    ├── auto
    │   ├── playbook_1
    │   ├── playbook_2
    │   └── playbook_n
    └── manual
        ├── playbook_1
        ├── playbook_2
        └── playbook_n

which would be run like:

Note

More options can be used, but they will mostly depend on the playbook’s functionality.

ansible-playbook -i /path/to/inventory \
                 /path/to/ansible/playbooks/manual/<playbook>

2.2. Ansible launcher

Using multiple environments, launching ansible from a non-standard location, among others may result in a large inconvinient command. Furthermore, if a run is to be triggered by an external entity, such as a script that requires ansible to run certain tasks within particular servers (see Section 1.3), additional concerns arise (e.g. What if I run ansible while it is already running? how to control the number of runs at a given time?).

To solve the abovementioned issues one can create a template ansible will render as a script:

#!/bin/bash
# Title       : run_ansible
# Description : Run ansible
# Author      : Tomas Felipe Llano Rios
# Date        : Nov 21, 2018
# Usage       : bash run_ansible [options]
# Help        : bash run_ansible -h
#==============================================================================

# tee only reads and prints from and to a file descriptor,
# so we need to use two execs to read and print from and to
# both stdout and stderr.
#
# Receives stdout, logs it and prints to stdout
exec > >(tee -ia /{{ ansible_log_dir }}/scheduled_run.log)
# Receive stderr, log it and print to stderr.
exec 2> >(tee -ia /{{ ansible_log_dir }}/scheduled_run.log >&2)

function log {
    echo "[$(date --rfc-3339=seconds)]: $*"
}

function print_help {
    echo -e "\nUsage: run_ansible [options]\n"
    echo -e "Where [options] include all ansible-playbook options,"
    echo -e "except for --vault-id and --inventory-file.\n"
    echo -e "Installed using the following configuration:"
    echo -e "\tEnvironment: $env"
    echo -e "\tAnsible home: $repo_dir"
    echo -e "\tAnsible config file: $cfg_file"
    echo -e "\tInventory file: $inv_file\n"

    command -v ansible-playbook > /dev/null 2>&1
    if [ "$?" -eq 0 ]; then
        echo -e "ansible-playbook options:\n"
        ansible-playbook -h | awk '/Options:/{y=1;next}y'
    else
        echo -e "See ansible-playbook help for more information.\n"
    fi
}

# Always release lock before exiting
function finish {
    flock -u 3
    rm -rf $lock
}
trap finish EXIT

# Create lock to prevent the script from being
# executed more than once at a given time.
declare -r script_name=`basename $0`
declare -r lock="/var/run/${script_name}"
if [ -f "$lock" ]; then
    echo "Another process (pid:`cat $lock`) is already running"
    trap - EXIT
    exit 1
fi
exec 3>$lock
flock -n 3
log "Lock acquired"
declare -r pid="$$"
echo "$pid" 1>&3

declare -r env={{ env }}
declare -r repo_dir={{ repo_dir }}
declare -r cfg_file="$repo_dir/ansible.cfg"
declare -r inv_file="$repo_dir/environments/$env/inventory"

if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
    print_help
    exit 0
fi

log "Updating repository"
cd $repo_dir
git pull
cd -

log "Executing ansible"

export ANSIBLE_CONFIG="$cfg_file"
export DEFAULT_ROLES_PATH="$repo_dir/roles"
export ANSIBLE_EXTRA_VARS="env=$env repo_dir=$repo_dir $ANSIBLE_EXTRA_VARS"
ansible-playbook --inventory-file "$inv_file" \
                 --extra-vars "$ANSIBLE_EXTRA_VARS" \
                 --vault-id "$env@$repo_dir/scripts/vault-secrets-client.sh" \
                 $repo_dir/site.yml \
                 -vv \
                 $@
unset ANSIBLE_CONFIG

In order to allow for easy migration of the script to, for example, another folder while still pointing to the project sources the template obtains one variable, ansible_log_dir, from group_vars and the remaining two from the bootstrap script. This is corroborated in line 47 from Section 3, where there are two extra vars (env, repo_dir) passed to ansible-playbook; all of which are dynamically discovered just before the first run.

On subsequent runs, run_ansible will keep passing down these values recursively. One could argue it is better to just place the task inside a one-shot playbook; this implies, however, that modifications made to the template should be applied manually and if the rendered script is changed it would not be restored automatically.

2.3. Scheduled run

It would be tedious to manually run ansible every time a change is done to the project. A nice approach to schedule when one wants provisioning to occur is creating a task to install a cron managing the time at which to call ansible:

Hint

Given the limited environment in which crons are run, one may need to add a task such as:

- name: Adding PATH variable to cronfile
  cron:
    name: PATH
    user: root
    env: yes
    value: /bin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
    cron_file: ansible_scheduled_run

or, if your launcher is written in bash, make it act as if it had been invoked as a login shell by using the -l option (#!/bin/bash -l).

- name: Schedule ansible to run every 30 minutes
  cron:
    name: "run ansible every 30 minutes"
    user: root
    minute: "*/30"
    job: "/path/to/run_ansible"
    cron_file: ansible_scheduled_run

3. bootstrap

By means of a few initial instructions the script should install ansible in the target system, prepare the environment it requires and trigger its first run. To further review whether the bootstrap failed or succeeded, and take apropriate actions, it is strongly recommended to log the output. Take the following program for example:

#!/bin/bash
# Title       : bootstrap.sh
# Description : Install and configure ansible
# Author      : Tomas Felipe Llano Rios
# Date        : Nov 21, 2018
# Usage       : bash bootstrap.sh <environment>
#==============================================================================

# tee only reads and prints from and to a file descriptor,
# so we need to use two execs to read and print from and to
# both stdout and stderr.
#
# Receive stdout, log it and print to stdout.
exec > >(tee -ia ./bootstrap_run.log)
# Receive stderr, log it and print to stderr.
exec 2> >(tee -ia ./bootstrap_run.log >&2)

declare -r env="$1"
declare -r script_path="$(readlink -e $0)"
declare -r script_dir="$(dirname $script_path)"
declare -r repo_dir="${script_dir%/*}"
declare -r cfg_file="$repo_dir/ansible.cfg"
declare -r inv_file="$repo_dir/environments/$env/inventory"

# Check if the environment provided exists within ansible's
# directory hierarchy.
declare -r envs="$(ls $repo_dir/environments/)"
if [ "$envs" != *"$env"* ]; then
    echo -e "\nUnrecognized environment. Choose from:\n$envs\n"
    exit 1
fi

# ansible-vault requires pycrypto 2.6, which is not installed by default
# on RHEL6 based systems.
declare -i centos_version=`rpm --query centos-release | awk -F'-' '{print $3}'`
if [ "$centos_version" -eq "6" ]; then
    /usr/bin/yum --enablerepo=epel -y install python-crypto2.6
fi
# Install ansible.
/usr/bin/yum --enablerepo=epel -y install ansible

# Run ansible.
export ANSIBLE_CONFIG="$cfg_file"
export DEFAULT_ROLES_PATH="$repo_dir/roles"
ansible-playbook \
    --inventory-file="$inv_file" \
    --extra-vars "env=$env repo_dir=$repo_dir" \
    --vault-id "$env@$repo_dir/scripts/vault-secrets-client.sh" \
    $repo_dir/site.yml \
    -vv

After running the script, there should be a cronfile in /etc/cron.d and a rendered version of the run_ansible script.

4. Example

1. Create directory tree

   cd /some/dir/
   mkdir -p ansible
   cd ansible && git init
   git remote add origin <uri>
   mkdir -p {playbooks,environments,roles,scripts}
   mkdir -p roles/master/{tasks,templates}
   mkdir -p environments/production/group_vars/
   # Create the appropriate files according to your needs.
   # A good start would be:
   #touch site.yml \ # Calls the master.yml playbook
   #      playbooks/master.yml \ # Calls the master role
   #      roles/master/tasks/main.yml \ # Renders template
   #      roles/master/templates/run_ansible.j2
   #      environment/production/inventory
   #      environment/production/group_vars/all
   

2. Download repo

   Note

   Consider using read-only deploy keys to download the repo without having to type an username and password; especially in unattended deployments.

   ssh <user>@<server>
   cd /usr/local/
   git clone <uri>
   

3. Bootstrap. Suppose you run the bootstrap script from /usr/local/ansible/scripts/, which discovers and passes two variables to ansible: env and repo_dir:

   declare -r env="$1"
   declare -r script_path="$(readlink -e $0)"
   declare -r script_dir="$(dirname $script_path)"
   declare -r repo_dir="${script_dir%/*}"
   declare -r cfg_file="$repo_dir/ansible.cfg"
   declare -r inv_file="$repo_dir/environments/$env/inventory"
   ansible-playbook \
       --inventory-file="$inv_file" \
       --extra-vars "env=$env repo_dir=$repo_dir" \
       --vault-id "$env@$repo_dir/scripts/vault-secrets-client.sh" \
       $repo_dir/site.yml \
       -vv
   

   Executing the script in a production environment, like bootstrap.sh prod, will cause variables to be passed to ansible as env=production and repo_dir=/usr/local/ansible/; therefore producing a run_ansible script pointing to /usr/local/ansible/.

4. Check for errors

   less bootstrap_run.log