1. Overview¶
The proposed architecture aims to provision a system by following the steps below:
- Create ansible’s directory hierarchy, playbooks, roles, taskfiles, etc. (basically,the logic that will be used to provision the system), and maintain it under version control.
- ssh into the master node and download the repo from version control. Consider using read-only deploy keys to download the repo without having to type an username and password; especially in unattended deployments.
- Run a one-shot script to perform an initial setup of the environment required for ansible to run properly during and after deployment, followed by the actual software execution (see Section 3).
- Check the bootstrap log and verify there were no errors and/or unexpected behaviors.
- If errors arose, fix them and re-run the bootstrap script.
2. Provisioning logic¶
2.1. Directory hierarchy¶
The logic should consider constant creation of new playbooks, roles, taskfiles, etc., allowing to easy scale in the future. Take the example below:
# tree /etc/ansible . ├── ansible.cfg ├── environments │ ├── dev │ │ ├── group_vars │ │ └── inventory │ └── prod │ ├── group_vars │ └── inventory ├── playbooks │ ├── playbook_1.yml │ ├── playbook_2.yml │ └── playbook_n.yml ├── roles │ ├── role_1 │ │ ├── handlers │ │ │ └── main.yml │ │ ├── tasks │ │ │ └── main.yml │ │ ├── templates │ │ └── vars │ │ └── main.yml │ ├── role_2 │ │ ├── handlers │ │ │ └── main.yml │ │ ├── tasks │ │ │ └── main.yml │ │ ├── templates │ │ └── vars │ │ └── main.yml │ └── role_n │ ├── handlers │ │ └── main.yml │ ├── tasks │ │ └── main.yml │ ├── templates │ └── vars │ └── main.yml ├── scripts │ ├── bootstrap.sh │ └── vault-client.sh └── site.yml
It is designed so that upon calling site.yml
tasks from a particular set of playbooks,
from the playbooks
folder, are applied. This behavior can be accomplished by
manually importing the playbooks or using variables:
Importing playbooks | Using variables |
# /etc/ansible/site.yml
---
- import_playbook: playbooks/playbook_1.yml
- import_playbook: playbooks/playbook_2.yml
- import_playbook: playbooks/playbook_n.yml
|
# /etc/ansible/environments/prod/group_vars/group1
---
playbook: playbook_1
# /etc/ansible/environments/prod/group_vars/group2
---
playbook: playbook_2
# /etc/ansible/environments/prod/group_vars/groupN
---
playbook: playbook_n
# /etc/ansible/site.yml
---
- import_playbook: "playbooks/{{ playbook }}.yml"
|
One could couple all playbooks inside site.yml
, but that would make future scalability difficult
and potentially cause problems if a large number of people is working on the same project
(take git merge conflicts for example).
If one-shot playbooks (playbooks that run only once, such as whose involving firmware updates) are to be managed,
it is recommended to modify the directory hierarchy so that the playbooks
folder holds
them. For example:
.
└── playbooks
├── auto
│ ├── playbook_1
│ ├── playbook_2
│ └── playbook_n
└── manual
├── playbook_1
├── playbook_2
└── playbook_n
which would be run like:
Note
More options can be used, but they will mostly depend on the playbook’s functionality.
ansible-playbook -i /path/to/inventory \
/path/to/ansible/playbooks/manual/<playbook>
2.2. Ansible launcher¶
Using multiple environments, launching ansible from a non-standard location, among others may result in a large inconvinient command. Furthermore, if a run is to be triggered by an external entity, such as a script that requires ansible to run certain tasks within particular servers (see Section 1.3), additional concerns arise (e.g. What if I run ansible while it is already running? how to control the number of runs at a given time?).
To solve the abovementioned issues one can create a template ansible will render as a script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | #!/bin/bash
# Title : run_ansible
# Description : Run ansible
# Author : Tomas Felipe Llano Rios
# Date : Nov 21, 2018
# Usage : bash run_ansible [options]
# Help : bash run_ansible -h
#==============================================================================
# tee only reads and prints from and to a file descriptor,
# so we need to use two execs to read and print from and to
# both stdout and stderr.
#
# Receives stdout, logs it and prints to stdout
exec > >(tee -ia /{{ ansible_log_dir }}/scheduled_run.log)
# Receive stderr, log it and print to stderr.
exec 2> >(tee -ia /{{ ansible_log_dir }}/scheduled_run.log >&2)
function log {
echo "[$(date --rfc-3339=seconds)]: $*"
}
function print_help {
echo -e "\nUsage: run_ansible [options]\n"
echo -e "Where [options] include all ansible-playbook options,"
echo -e "except for --vault-id and --inventory-file.\n"
echo -e "Installed using the following configuration:"
echo -e "\tEnvironment: $env"
echo -e "\tAnsible home: $repo_dir"
echo -e "\tAnsible config file: $cfg_file"
echo -e "\tInventory file: $inv_file\n"
command -v ansible-playbook > /dev/null 2>&1
if [ "$?" -eq 0 ]; then
echo -e "ansible-playbook options:\n"
ansible-playbook -h | awk '/Options:/{y=1;next}y'
else
echo -e "See ansible-playbook help for more information.\n"
fi
}
# Always release lock before exiting
function finish {
flock -u 3
rm -rf $lock
}
trap finish EXIT
# Create lock to prevent the script from being
# executed more than once at a given time.
declare -r script_name=`basename $0`
declare -r lock="/var/run/${script_name}"
if [ -f "$lock" ]; then
echo "Another process (pid:`cat $lock`) is already running"
trap - EXIT
exit 1
fi
exec 3>$lock
flock -n 3
log "Lock acquired"
declare -r pid="$$"
echo "$pid" 1>&3
declare -r env={{ env }}
declare -r repo_dir={{ repo_dir }}
declare -r cfg_file="$repo_dir/ansible.cfg"
declare -r inv_file="$repo_dir/environments/$env/inventory"
if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
print_help
exit 0
fi
log "Updating repository"
cd $repo_dir
git pull
cd -
log "Executing ansible"
export ANSIBLE_CONFIG="$cfg_file"
export DEFAULT_ROLES_PATH="$repo_dir/roles"
export ANSIBLE_EXTRA_VARS="env=$env repo_dir=$repo_dir $ANSIBLE_EXTRA_VARS"
ansible-playbook --inventory-file "$inv_file" \
--extra-vars "$ANSIBLE_EXTRA_VARS" \
--vault-id "$env@$repo_dir/scripts/vault-secrets-client.sh" \
$repo_dir/site.yml \
-vv \
$@
unset ANSIBLE_CONFIG
|
In order to allow for easy migration of the script to, for example, another folder while still pointing
to the project sources the template obtains one variable, ansible_log_dir, from group_vars
and the remaining two from the bootstrap script. This is corroborated
in line 47 from Section 3, where there are two extra vars (env, repo_dir)
passed to ansible-playbook
; all of which are dynamically discovered just before the first run.
On subsequent runs, run_ansible
will keep passing down these values recursively. One could argue it is better
to just place the task inside a one-shot playbook; this implies, however, that modifications made to the template
should be applied manually and if the rendered script is changed it would not be restored automatically.
2.3. Scheduled run¶
It would be tedious to manually run ansible every time a change is done to the project. A nice approach to schedule when one wants provisioning to occur is creating a task to install a cron managing the time at which to call ansible:
Hint
Given the limited environment in which crons are run, one may need to add a task such as:
- name: Adding PATH variable to cronfile
cron:
name: PATH
user: root
env: yes
value: /bin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
cron_file: ansible_scheduled_run
or, if your launcher is written in bash, make it act as if it had been invoked as a login shell
by using the -l
option (#!/bin/bash -l
).
- name: Schedule ansible to run every 30 minutes
cron:
name: "run ansible every 30 minutes"
user: root
minute: "*/30"
job: "/path/to/run_ansible"
cron_file: ansible_scheduled_run
3. bootstrap¶
By means of a few initial instructions the script should install ansible in the target system, prepare the environment it requires and trigger its first run. To further review whether the bootstrap failed or succeeded, and take apropriate actions, it is strongly recommended to log the output. Take the following program for example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | #!/bin/bash
# Title : bootstrap.sh
# Description : Install and configure ansible
# Author : Tomas Felipe Llano Rios
# Date : Nov 21, 2018
# Usage : bash bootstrap.sh <environment>
#==============================================================================
# tee only reads and prints from and to a file descriptor,
# so we need to use two execs to read and print from and to
# both stdout and stderr.
#
# Receive stdout, log it and print to stdout.
exec > >(tee -ia ./bootstrap_run.log)
# Receive stderr, log it and print to stderr.
exec 2> >(tee -ia ./bootstrap_run.log >&2)
declare -r env="$1"
declare -r script_path="$(readlink -e $0)"
declare -r script_dir="$(dirname $script_path)"
declare -r repo_dir="${script_dir%/*}"
declare -r cfg_file="$repo_dir/ansible.cfg"
declare -r inv_file="$repo_dir/environments/$env/inventory"
# Check if the environment provided exists within ansible's
# directory hierarchy.
declare -r envs="$(ls $repo_dir/environments/)"
if [ "$envs" != *"$env"* ]; then
echo -e "\nUnrecognized environment. Choose from:\n$envs\n"
exit 1
fi
# ansible-vault requires pycrypto 2.6, which is not installed by default
# on RHEL6 based systems.
declare -i centos_version=`rpm --query centos-release | awk -F'-' '{print $3}'`
if [ "$centos_version" -eq "6" ]; then
/usr/bin/yum --enablerepo=epel -y install python-crypto2.6
fi
# Install ansible.
/usr/bin/yum --enablerepo=epel -y install ansible
# Run ansible.
export ANSIBLE_CONFIG="$cfg_file"
export DEFAULT_ROLES_PATH="$repo_dir/roles"
ansible-playbook \
--inventory-file="$inv_file" \
--extra-vars "env=$env repo_dir=$repo_dir" \
--vault-id "$env@$repo_dir/scripts/vault-secrets-client.sh" \
$repo_dir/site.yml \
-vv
|
After running the script, there should be a cronfile in /etc/cron.d and a rendered version of the run_ansible script.
4. Example¶
Create directory tree
cd /some/dir/ mkdir -p ansible cd ansible && git init git remote add origin <uri> mkdir -p {playbooks,environments,roles,scripts} mkdir -p roles/master/{tasks,templates} mkdir -p environments/production/group_vars/ # Create the appropriate files according to your needs. # A good start would be: #touch site.yml \ # Calls the master.yml playbook # playbooks/master.yml \ # Calls the master role # roles/master/tasks/main.yml \ # Renders template # roles/master/templates/run_ansible.j2 # environment/production/inventory # environment/production/group_vars/all
Download repo
Note
Consider using read-only deploy keys to download the repo without having to type an username and password; especially in unattended deployments.
ssh <user>@<server> cd /usr/local/ git clone <uri>
Bootstrap. Suppose you run the bootstrap script from
/usr/local/ansible/scripts/
, which discovers and passes two variables to ansible: env and repo_dir:1 2 3 4 5 6 7 8 9 10 11 12
declare -r env="$1" declare -r script_path="$(readlink -e $0)" declare -r script_dir="$(dirname $script_path)" declare -r repo_dir="${script_dir%/*}" declare -r cfg_file="$repo_dir/ansible.cfg" declare -r inv_file="$repo_dir/environments/$env/inventory" ansible-playbook \ --inventory-file="$inv_file" \ --extra-vars "env=$env repo_dir=$repo_dir" \ --vault-id "$env@$repo_dir/scripts/vault-secrets-client.sh" \ $repo_dir/site.yml \ -vv
Executing the script in a production environment, like
bootstrap.sh prod
, will cause variables to be passed to ansible asenv=production
andrepo_dir=/usr/local/ansible/
; therefore producing arun_ansible
script pointing to/usr/local/ansible/
.Check for errors
less bootstrap_run.log