Ansible Up and Running, 2nd Edition

这本书其实挺不错的，我还总结了在工作中ansible的常用模块在另一个blog里面。

Could we remove major architectural components from the IT automation stack? Eliminating management daemons and relying instead on OpenSSH meant the system could start managing a computer fleet immediately, without having to set up anything on the managed machines.

the “Making Ansible Go Even Faster” chapter now covers asynchronous tasks, and the “Debugging Ansible Playbooks” chapter now covers the debugger that was introduced in version 2.1.

we are all slowly turning into system engineers.

Chapter 1. Introduction

When we talk about configuration management, we are typically talking about writing some kind of state description for our servers, and then using a tool to enforce that the servers are, indeed, in that state: the right packages are installed, configuration files contain the expected values and have the expected permissions, the right services are running, and so on.

Ansible is a great tool for deployment as well as configuration management. Using a single tool for both configuration management and deployment makes life simpler for the folks responsible for operations.

Some people talk about the need for orchestration of deployment. This is where multiple remote servers are involved, and things have to happen in a specific order.

How Ansible works

In Ansible, a script is called a playbook. A playbook describes which hosts (what Ansible calls remote servers) to configure, and an ordered list of tasks to perform on those hosts.

Ansible will make SSH connections in parallel to web1, web2, and web3. It will execute the first task on the list on all three hosts simultaneously

To manage a remote server with Ansible, the server needs to have SSH and Python 2.5 or later installed, or Python 2.4 with the Python simplejsonlibrary installed. There’s no need to preinstall an agent or anyother software on the host.

The control machine (the one that you use to control remote machines) needs to have Python 2.6 or later installed.

Ansible is push based, and has been used successfully in production with thousands of nodes, and has excellent support for environments where servers are dynamically added and removed.

Ansible modules are declarative; you use them to describe the state you want the server to be in. Modules are also idempotent.

Ansible has excellent support for templating, as well as defining variables at different scopes. Anybody who thinks Ansible is equivalent to working with shell scripts has never had to maintain a nontrivial program written in shell. I’ll always choose Ansible over shell scripts for config management tasks if given a choice.

To be productive with Ansible, you need to be familiar with basic Linux system administration tasks. Ansible makes it easy to automate your tasks, but it’s not the kind of tool that “automagically” does things that you otherwise wouldn’t know how to do.

Ansible uses the YAML file format and the Jinja2 templating languages, so you’ll need to learn some YAML and Jinja2 to use Ansible, but both technologies are easy to pick up.

If you prefer not to spend the money on a public cloud, I recommend you install Vagrant on your machine. Vagrant is an excellent open source tool for managing virtual machines. You can use Vagrant to boot a Linux virtual machine inside your laptop, and you can use that as a test server.

Vagrant needs the VirtualBox virtualizer to be installed on your machine. Download VirtualBox and then download Vagrant.

mkdir playbooks
cd playbooks
vagrant init ubuntu/trusty64
vagrant up

The first time you use vagrant up, it will download the virtual machine image file, which might take a while, depending on your internet connection.

You should be able to SSH into your new Ubuntu 14.04 virtual machine by running the following:

vagrant ssh

This approach lets us interact with the shell, but Ansible needs to connect to the virtual machine by using the regular SSH client, not the vagrant ssh command.

Tell Vagrant to output the SSH connection details by typing the following:

vagrant ssh-config

ssh vagrant@127.0.0.1 -p 2222 -i /Users/lorin/dev/ansiblebook/ch01/
playbooks/.vagrant/machines/default/virtualbox/private_key

testserver ansible_host=127.0.0.1 ansible_port=2222 ansible_user=vagrant ansible_private_key_file=.vagrant/machines/default/virtualbox/private_key

~/.ansible.cfg

Ansible supports the ssh-agent program, so you don’t need to explicitly specify SSH key files in your inventory files. See “SSH Agent” for more details if you haven’t used ssh-agent before

If Ansible did not succeed, add the -vvvv flag to see more details about the error:

ansible testserver -i hosts -m ping -vvvv

Simplify by ansible.cfg file

we’ll use one such mechanism, the ansible.cfg file, to set some defaults so we don’t need to type as much.

Ansible looks for an ansible.cfg file in the following places, in this order:

• File specified by the ANSIBLE_CONFIG environment variable
• ./ansible.cfg (ansible.cfg in the current directory)
• ~/.ansible.cfg (.ansible.cfg in your home directory)
• /etc/ansible/ansible.cfg

I typically put ansible.cfg in the current directory, alongside my playbooks. That way, I can check it into the same version-control repository that my playbooks are in.

[defaults]
inventory = hosts
remote_user = vagrant
private_key_file = .vagrant/machines/default/virtualbox/private_key
host_key_checking = False

Disables SSH host-key checking. Otherwise, we need to edit our ~/.ssh/known_hosts file every time we destroy and re-create a nodes.

Ansible uses /etc/ansible/hosts as the default location for the inventory file. However, I never use this because I like to keep my inventory files version-controlled alongside my playbooks.

The command module is so commonly used that it’s the default module, so we can omit it

ansible testserver -a uptime
## spaces in command use quotes
ansible testserver -a "tail /var/log/dmesg"
## -b becomes root user
ansible testserver -b -a "tail /var/log/syslog"

Chapter 2. Playbooks: A Beginning

Most of your time in Ansible will be spent writing playbooks. A playbook is the term that Ansible uses for a configuration management script.

vargant virtual machine ports mapping in vagrantfile:

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.network "forwarded_port", guest: 80, host: 8080
  config.vm.network "forwarded_port", guest: 443, host: 8443
end

vagrant reload

==> default: Forwarding ports...
    default: 80 => 8080 (adapter 1)
    default: 443 => 8443 (adapter 1)
    default: 22 => 2222 (adapter 1)

TLS VERSUS SSL

You might be familiar with the term SSL rather than TLS(Transport Layer Security) in the context of secure web servers. SSL is an older protocol that was used to secure communications between browsers and web servers, and it has been superseded by a newer protocol named TLS. Although many continue to use the term SSL to refer to the current secure protocol, in this book, I use the more accurate TLS.

WHY DO YOU USE TRUE IN ONE PLACE AND YES IN ANOTHER?

Strictly speaking, module arguments (for example, update_cache=yes) are treated differently from values elsewhere in playbooks (for example, sudo: True). Values elsewhere are handled by the YAML parser and so use the YAML conventions of truthiness:

YAML truthy

true, True, TRUE, yes, Yes, YES, on, On, ON, y, Y

YAML falsey

false, False, FALSE, no, No, NO, off, Off, OFF, n, N

Module arguments are passed as strings and use Ansible’s internal conventions:

module arg truthy

yes, on, 1, true

module arg falsey

no, off, 0, false

I tend to follow the examples in the official Ansible documentation. These typically use yes and no when passing arguments to modules (since that’s consistent with the module documentation), and True and False elsewhere in playbooks.

NOTE

An Ansible convention is to keep files in a subdirectory named files, and Jinja2 templates in a subdirectory named templates. I follow this convention throughout the book.

For .j2 file, when Ansible renders this template, it will replace this variable with the real value.

Inventory files are in the .ini file format.

COWSAY

If you have the cowsay program installed on your local machine, Ansible output will look like this instead:

you can download cowsay rpm and install it:

cowsay -l

Cow files in /usr/share/cowsay:
beavis.zen blowfish bong bud-frogs bunny cheese cower default dragon
dragon-and-cow elephant elephant-in-snake eyes flaming-sheep ghostbusters
head-in hellokitty kiss kitty koala kosh luke-koala mech-and-cow meow milk
moofasa moose mutilated ren satanic sheep skeleton small sodomized
stegosaurus stimpy supermilker surgery telebears three-eyes turkey turtle
tux udder vader vader-koala www

set what animal you like:

export ANSIBLE_COW_SELECTION=tux

enable cowsay:

export ANSIBLE_NOCOWS=0

then if you run playbook:

< PLAY [Configure webserver with nginx] >
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

If you don’t want to see the cows, you can disable cowsay by setting the ANSIBLE_NOCOWS environment variable like this:

export ANSIBLE_NOCOWS=1

You can also disable cowsay by adding the following to your ansible.cfg file:

[defaults]
nocows = 1

TIP

If your playbook file is marked as executable and starts with a line that looks like this

#!/usr/bin/env ansible-playbook

then you can execute it by invoking it directly, like this:

./playbook-file.yml

YAML syntax

Start of file

---

However, if you forget to put those three dashes at the top of your playbook files, Ansible won’t complain.

String

In general, YAML strings don’t have to be quoted, although you can quote them if you prefer. Even if there are spaces, you don’t need to quote them.

In some scenarios in Ansible, you will need to quote strings. These typically involve the use of for variable substitution.

Boolean

YAML has a native Boolean type, and provides you with a wide variety of strings that can be interpreted as true or false.

List

- My Fair Lady
- Oklahoma
- The Pirates of Penzance

Dictionary

address: 742 Evergreen Terrace
city: Springfield
state: North Takoma

Line folding

When writing playbooks, you’ll often encounter situations where you’re passing many arguments to a module. For aesthetics, you might want to break this up across multiple lines in your file, but you want Ansible to treat the string as if it were a single line.

You can do this with YAML by using line folding with the greater than (>) character. The YAML parser will replace line breaks with spaces. For example:

address: >
    Department of Computer Science,
    A.V. Williams Building,
    University of Maryland
city: College Park
state: Maryland

Anatomy of a Playbook

A playbook is a list of plays:

---
- name: Configure webserver with nginx
  hosts: webservers
  become: True
  tasks:
  - name: install nginx
      apt: name=nginx update_cache=yes

  - name: copy nginx config file
    copy: src=files/nginx.conf dest=/etc/nginx/sites-available/default

  - name: enable configuration
    file: >
      dest=/etc/nginx/sites-enabled/default
      src=/etc/nginx/sites-available/default
      state=link

  - name: copy index.html
    template: src=templates/index.html.j2  dest=/usr/share/nginx/html/index.html mode=0644

  - name: restart nginx
    service: name=nginx state=restarted

you’ll see in Chapter 16, you can use the --start-at-task <task name> flag to tell ansible-playbook to start a playbook in the middle of a play, but you need to reference the task by name.

Every task must contain a key with the name of a module and a value with the arguments to that module. In the preceding example, the module name is apt and the arguments are name=nginx update_cache=yes.

The arguments are treated as a string, not as a dictionary. This means that if you want to break arguments into multiple lines, you need to use the YAML folding syntax, like this:

- name: install nginx
  apt: >
      name=nginx
      update_cache=yes

Ansible also supports a task syntax that will let you specify module arguments as a YAML dictionary, which is helpful when using modules that support complex arguments. For example:

- name: Install docker
  any_errors_fatal: true
  yum:
    name: docker-ce
    state: present
    enablerepo: docker-local

Modules

Modules are scripts that come packaged with Ansible and perform some kind of action on a host.

commonly used modules:

• yum
• copy
• file
• service
• template

VIEWING ANSIBLE MODULE DOCUMENTATION

Ansible ships with the ansible-doc command-line tool, which shows documentation about modules. Think of it as man pages for Ansible modules.

ansible-doc yum

Recall from the first chapter that Ansible executes a task on a host by generating a custom script based on the module name and arguments, and then copies this script to the host and runs it.

More than 200 modules ship with Ansible, and this number grows with every release. You can also find third-party Ansible modules out there, or write your own.

Putting It All Together

To sum up, a playbook contains one or more plays. A play associates an unordered set of hosts with an ordered list of tasks. Each task is associated with exactly one module.

+-------------+         +------------+          +-------------+
|             |     +--->            |      +--->             |
|  playbook   +--------->    play    +---------->    hosts    |
|             |     +--->            |      +--->             |
+-------------+         +------+-----+          +-------------+
                               |
                               |
                               |
                             +---+
                             | | |
                        +----v-v-v----+         +-------------+
                        |             |         |             |
                        |   task      +-------->+   module    |
                        |             |         |             |
                        +-------------+         +-------------+

Did Anything Change? Tracking Host State

Ansible modules will first check to see whether the state of the host needs to be changed before taking any action. If the state of the host matches the arguments of the module, Ansible takes no action on the host and responds with a state of ok.

Ansible’s detection of state change can be used to trigger additional actions through the use of handlers. But, even without using handlers, it is still a useful form of feedback to see whether your hosts are changing state as the playbook runs.

Variables and Handlers

- name: Configure webserver with nginx and tls
  hosts: webservers
  become: True
  vars:
    key_file: /etc/nginx/ssl/nginx.key
    cert_file: /etc/nginx/ssl/nginx.crt
    conf_file: /etc/nginx/sites-available/default
    server_name: localhost
  tasks:
    - name: Install nginx
      apt: name=nginx update_cache=yes cache_valid_time=3600

    - name: create directories for ssl certificates
      file: path=/etc/nginx/ssl state=directory

    - name: copy TLS key
      copy: src=files/nginx.key dest={{ key_file }} owner=root mode=0600
      notify: restart nginx

    - name: copy TLS certificate
      copy: src=files/nginx.crt dest={{ cert_file }}
      notify: restart nginx

    - name: copy nginx config file
      template: src=templates/nginx.conf.j2 dest={{ conf_file }}
      notify: restart nginx

    - name: enable configuration
      file: dest=/etc/nginx/sites-enabled/default src={{ conf_file }} state=link
      notify: restart nginx

    - name: copy index.html
      template: src=templates/index.html.j2 dest=/usr/share/nginx/html/index.html
             mode=0644

  handlers:
    - name: restart nginx
      service: name=nginx state=restarted

Generating a TLS Certificate

In a production environment, you’d purchase your TLS certificate from a certificate authority, or use a free service such as Let’s Encrypt, which Ansible supports via the letsencrypt module.

Here we use self-signed certificate generated free of charge:

mkdir files
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
    -subj /CN=localhost \
    -keyout files/nginx.key -out files/nginx.crt

Any valid YAML can be used as the value of a variable. You can use lists and dictionaries in addition to strings and Booleans.

Variables

Variables can be used in tasks, as well as in template files. You reference variables by using the  notation. Ansible replaces these braces with the value of the variable.

WHEN QUOTING IS NECESSARY

bad:

- name: perform some task
  command: {{ myapp }} -a foo

good:

- name: perform some task
  command: "{{ myapp }} -a foo"

A similar problem arises if your argument contains a colon. For example, bad:

- name: show a debug message
  debug: msg="The debug module will print a message: neat, eh?"

good:

- name: show a debug message
  debug: "msg='The debug module will print a message: neat, eh?'"

Generating the Template

We put templates in templates folder, we use the .j2 extension to indicate that the file is a Jinja2 template. However, you can use a different extension if you like; Ansible doesn’t care.

You can use all of the Jinja2 features in your templates, you probably won’t need to use those advanced templating features, though. One Jinja2 feature you probably will use with Ansible is filters: Jinja2 Template Designer Documentation.

Handlers

Handlers are one of the conditional forms that Ansible supports. A handler is similar to a task, but it runs only if it has been notified by a task. A task will fire the notification if Ansible recognizes that the task has changed the state of the system. A task notifies a handler by passing the handler’s name as the argument.

A FEW THINGS TO KEEP IN MIND ABOUT HANDLERS

Handlers usually run after all of the tasks are run at the end of the play. They run only once, even if they are notified multiple times. If a play contains multiple handlers, the handlers always run in the order that they are defined in the handlers section, not the notification order.

The official Ansible docs mention that the only common uses for handlers are for restarting services and for reboots. Personally, I’ve always used them only for restarting services.Even then, it’s a pretty small optimization, since we can always just unconditionally restart the service at the end of the playbook instead of notifying it on change, and restarting a service doesn’t usually take very long.

Chapter 3. Inventory: Describing Your Servers

The collection of hosts that Ansible knows about is called the inventory. In this chapter, you will learn how to describe a set of hosts as an Ansible inventory.

Ansible automatically adds one host to the inventory by default: localhost. Ansible understands that localhost refers to your local machine, so it will interact with it directly rather than connecting by SSH.

Preliminaries: Multiple Vagrant Machines

Before you modify your existing Vagrantfile, make sure you destroy your existing virtual machine by running the following:

vagrant destroy --force

vagrantfile for three servers:

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # Use the same key for each machine
  config.ssh.insert_key = false

  config.vm.define "vagrant1" do |vagrant1|
    vagrant1.vm.box = "ubuntu/trusty64"
    vagrant1.vm.network "forwarded_port", guest: 80, host: 8080
    vagrant1.vm.network "forwarded_port", guest: 443, host: 8443
  end
  config.vm.define "vagrant2" do |vagrant2|
    vagrant2.vm.box = "ubuntu/trusty64"
    vagrant2.vm.network "forwarded_port", guest: 80, host: 8081
    vagrant2.vm.network "forwarded_port", guest: 443, host: 8444
  end
  config.vm.define "vagrant3" do |vagrant3|
    vagrant3.vm.box = "ubuntu/trusty64"
    vagrant3.vm.network "forwarded_port", guest: 80, host: 8082
    vagrant3.vm.network "forwarded_port", guest: 443, host: 8445
  end
end

Using the same key on each host simplifies our Ansible setup because we can specify a single SSH key in the ansible.cfg file:

[defaults]
inventory = inventory
remote_user = vagrant
private_key_file = ~/.vagrant.d/insecure_private_key
host_key_checking = False

In the inventory file, you can use ansible_host to explicitly specify IP and ansible_port indicates SSH port number:

vagrant1 ansible_host=127.0.0.1 ansible_port=2222
vagrant2 ansible_host=127.0.0.1 ansible_port=2200
vagrant3 ansible_host=127.0.0.1 ansible_port=2201

Behavioral Inventory Parameters

Markdown table generator

|            Name            	|     Default     	|                                   Description                                   	|
|:--------------------------:	|:---------------:	|:-------------------------------------------------------------------------------:	|
| ansible_host               	| Name of host    	| Hostname or IP address to SSH to                                                	|
| ansible_port               	| 22              	| Port to SSH to                                                                  	|
| ansible_user               	| Root            	| User to SSH as                                                                  	|
| ansible_password           	| (None)          	| Password to use for SSH authentication                                          	|
| ansible_connection         	| smart           	| How Ansible will connect to host (see the following section)                    	|
| ansible_private_key_file   	| (None)          	| SSH private key to use for SSH authenticatio                                    	|
| ansible_shell_type         	| sh              	| Shell to use for commands (see the following section)                           	|
| ansible_python_interpreter 	| /usr/bin/python 	| Python interpreter on host (see the following section)                          	|
| ansible_*_interpreter      	| (None)          	| Like ansible_python_interpreter for other languages (see the following section) 	|

Explanation:

• ansible_connection Ansible supports multiple transports, which are mechanisms that Ansible uses to connect to the host. The default transport, smart, will check whether the locally installed SSH client supports a feature called ControlPersist. If the SSH client supports ControlPersist, Ansible will use the local SSH client. If the SSH client doesn’t support ControlPersist, the smart transport will fall back to using a Python-based SSH client library called Paramiko.

• ansible_shell_type Ansible works by making SSH connections to remote machines and then invoking scripts. By default, Ansible assumes that the remote shell is the Bourne shell located at /bin/sh, and will generate the appropriate command-line parameters that work with Bourne shell.

  Ansible also accepts csh, fish, and (on Windows) powershell as valid values for this parameter. I’ve never encountered a need for changing the shell type.

• ansible_python_interpreter Because the modules that ship with Ansible are implemented in Python 2, Ansible needs to know the location of the Python interpreter on the remote machine. You might need to change this if your remote host does not have a Python 2 interpreter at /usr/bin/python. For example, if you are managing hosts that run Arch Linux, you will need to change this to /usr/bin/python2, because Arch Linux installs Python 3 at /usr/bin/python, and Ansible modules are not (yet) compatible with Python 3.

• ansible_*_interpreter If you are using a custom module that is not written in Python, you can use this parameter to specify the location of the interpreter (e.g., /usr/bin/ruby).

Note: You can override some of the behavioral parameter default values in the defaults section of the ansible.cfg file

| Behavioral inventory parameter 	| ansible.cfg option 	|
|:------------------------------:	|:------------------:	|
| ansible_port                   	| remote_port        	|
| ansible_user                   	| remote_user        	|
| ansible_private_key_file       	| private_key_file   	|
| ansible_shell_type             	| executable         	|

Groups

Ansible automatically defines a group called all (or *), which includes all of the hosts in the inventory. For example:

ansible all -a "date"
ansible '*' -a "date"

We can define our own groups in the inventory file. Ansible uses the .ini file format for inventory files. In the .ini format, configuration values are grouped together into sections. For example:

[master]

[workers]

[nodes]

[all:vars]
ansible_connection=ssh
ansible_user=root
ansible_ssh_private_key_file=null
gather_facts=True
gathering=smart
host_key_checking=False

Alias and Ports:

In inventory file:

[master]
<hostname>
<hostname>:<port number>
<alias> ansible_host=<IP> ansible_port=<port number>

Groups of groups

Ansible also allows you to define groups that are made up of other groups. Here web and task are groups, diango subgroup wrap them up.

[django:children]
web
task

Numbered hosts

[host]
web[1:20].example.com
## leading 0
web[01:20].example.com
web-[a-t].example.com

Host and Group Variables

Format: [:vars]

[all:vars]

[production:vars]

[staging:vars]

Additionally, though Ansible variables can hold Booleans, strings, lists, and dictionaries, in an inventory file, you can specify only Booleans and strings.

Ansible offers a more scalable approach to keep track of host and group variables: you can create a separate variable file for each host and each group. Ansible expects these variable files to be in YAML format.

Ansible looks for host variable files in a directory called host_vars and group variable files in a directory called group_vars. Ansible expects these directories to be either in the directory that contains your playbooks or in the directory adjacent to your inventory file. For example:

playbooks folder:
  - playbook.yml
  - group_vars folder
    - production

If we use YAML dictionary format in group variable file:

db:
    user: widgetuser
    password: pFmMxcyD;Fc6)6
    name: widget_production
    primary:
        host: rhodeisland.example.com
        port: 5432
    replica:
        host: virginia.example.com
        port: 5432

rabbitmq:
    host: pennsylvania.example.com
    port: 5672

when reference in playbook:

{{ db.primary.host }}

Dynamic Inventory

If the inventory file is marked executable, Ansible will assume it is a dynamic inventory script and will execute the file instead of reading it.

If some other system, such as AWS EC2, will keep track of the virtual machine information for us, we don’t necessarily need to write the inventory file manually, we can use dynamic inventory script to query about which machines are running and use them.

preexisting inventory scripts

Adding Entries at Runtime with add_host and group_by

Ansible will let you add hosts and groups to the inventory during the execution of a playbook. not use yet

Chapter 4. Variables and Facts

Ansible is not a full-fledged programming language, but it does have several programming language features, and one of the most important of these is variable substitution. This chapter presents Ansible’s support for variables in more detail, including a certain type of variable that Ansible calls a fact.

Defining Variables in Playbooks

There are two scenarios, for example:

- name: Configure Kubeadm
  hosts: master
  become: true
  any_errors_fatal: true
  vars:
    NODE_COUNT: "{{ groups['nodes'] | length }}"
  roles:
    - setup.master

## or
- name: Configure Kubeadm
  hosts: master
  become: true
  any_errors_fatal: true
  vars_files:
    - config.yml
  roles:
    - setup.master

Viewing the Values of Variables

We use the debug module to print out an arbitrary message. We can also use it to output the value of the variable.

- debug: var=<myvarname>

Registering Variables

Often, you’ll find that you need to set the value of a variable based on the result of a task.To do so, we create a registered variable using the register clause when invoking a module. Below shows how to capture the output of the whoami command to a variable named login.

- name: capture output of whoami command
  command: whoami
  register: login

Note, if you want to use login variable registered here, it’s not like that you can call it as

In order to use the login variable later, we need to know the type of value to expect. The value of a variable set using the register clause is always a dictionary, but the specific keys of the dictionary are different, depending on the module that was invoked.

Unfortunately, the official Ansible module documentation doesn’t contain information about what the return values look like for each module. The module docs do often contain examples that use the register clause, which can be helpful. I’ve found the simplest way to find out what a module returns is to register a variable and then output that variable with the debug module.

- name: show return value of command module
  hosts: server1
  tasks:
    - name: capture output of id command
      command: id -un
      register: login
    - debug: var=login

The shell module has the same output structure as the command module, but other modules contain different keys, the output here is:

TASK: [debug var=login] *******************************************************
ok: [server1] => {
    "login": {
        "changed": true, 1
        "cmd": [ 2
            "id",
            "-un"
        ],
        "delta": "0:00:00.002180",
        "end": "2015-01-11 15:57:19.193699",
        "invocation": {
            "module_args": "id -un",
            "module_name": "command"
        },
        "rc": 0, 3
        "start": "2015-01-11 15:57:19.191519",
        "stderr": "", 4
        "stdout": "vagrant", 5
        "stdout_lines": [ 6
            "vagrant"
        ],
        "warnings": []
    }
}

• The changed key is present in the return value of all Ansible modules, and Ansible uses it to determine whether a state change has occurred. For the command and shellmodule, this will always be set to true unless overridden with the changed_whenclause.

• The cmd key contains the invoked command as a list of strings.

• The rc key contains the return code. If it is nonzero, Ansible will assume the task failed to execute.

• The stdout key contains any text written to standard out, as a single string.

• The stdout_lines key contains any text written to split by newline. It is a list, and each element of the list is a line of output.

So now you can access login with:

- name: capture output of id command
  command: id -un
  register: login
- debug: msg="Logged in as user {{ login.stdout }}"

ACCESSING DICTIONARY KEYS IN A VARIABLE

If a variable contains a dictionary, you can access the keys of the dictionary by using either a dot (.) or a subscript ([]).

{{ login.stdout }}
{{ login['stdout'] }}

ansible_eth1['ipv4']['address']
ansible_eth1['ipv4'].address
ansible_eth1.ipv4['address']
ansible_eth1.ipv4.address

CAUTION

If your playbooks use registered variables, make sure you know the content of those variables, both for cases where the module changes the host’s state and for when the module doesn’t change the host’s state. Otherwise, your playbook might fail when it tries to access a key in a registered variable that doesn’t exist.

Facts

As you’ve already seen, when Ansible runs a playbook, before the first task runs, this happens:

GATHERING FACTS **************************************************
ok: [servername]

When Ansible gathers facts, it connects to the host and queries it for all kinds of details about the host: CPU architecture, operating system, IP addresses, memory info, disk info, and more. This information is stored in variables that are called facts, and they behave just like any other variable. For example:

- name: print out operating system
  hosts: all
  gather_facts: True
  tasks:
  - debug: var=ansible_distribution

List of facts variable can be found here

Viewing All Facts Associated with a Server

Ansible implements fact collecting through the use of a special module called the setupmodule. You don’t need to call this module in your playbooks because Ansible does that automatically when it gathers facts.

ansible server1 -m setup

server1 | success >> {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "10.0.2.15",
            "192.168.4.10"
        ],
        "ansible_all_ipv6_addresses": [
            "fe80::a00:27ff:fefe:1e4d",
            "fe80::a00:27ff:fe67:bbf3"
        ],
(many more facts)

Note that the returned value is a dictionary whose key is ansible_facts and whose value is a dictionary that contains the name and value of the actual facts.

Viewing a Subset of Facts

The setup module supports a filter parameter that lets you filter by fact name by specifying a glob.

ansible web -m setup -a 'filter=ansible_eth*'

Any Module Can Return Facts

The use of ansible_facts in the return value is an Ansible idiom. If a module returns a dictionary that contains ansible_facts as a key, Ansible will create variable names in the environment with those values and associate them with the active host.

For modules that return facts, there’s no need to register variables, since Ansible creates these variables for you automatically.

- name: get ec2 facts
  ec2_facts:

- debug: var=ansible_ec2_instance_id

Several modules ship with Ansible that return facts. You’ll see another one of them, the docker module.

Local Facts

Ansible provides an additional mechanism for associating facts with a host. You can place one or more files on the remote host machine in the /etc/ansible/facts.d directory. not used yet

Using set_fact to Define a New Variable

Ansible also allows you to set a fact (effectively the same as defining a new variable) in a task by using the set_fact module. I often like to use set_fact immediately after register to make it simpler to refer to a variable.

- name: get snapshot id
  shell: >
    aws ec2 describe-snapshots --filters
    Name=tag:Name,Values=my-snapshot
    | jq --raw-output ".Snapshots[].SnapshotId"
  register: snap_result

- set_fact: snap={{ snap_result.stdout }}

- name: delete old snapshot
  command: aws ec2 delete-snapshot --snapshot-id "{{ snap }}"

Built-in Variables

Ansible defines several variables that are always available in a playbook

| Parameter                	| Description                                                                                                                                                                                 	|
|--------------------------	|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------	|
| hostvars                 	| A dict whose keys are Ansible hostnames and values are dicts  that map variable names to values                                                                                             	|
| inventory_hostname       	| Fully qualified domain name of the current host as known by Ansible  (e.g., myhost.example.com)                                                                                             	|
| inventory_hostname_short 	| Name of the current host as known by Ansible, without the domain name  (e.g., myhost)                                                                                                       	|
| group_names              	| A list of all groups that the current host is a member of                                                                                                                                   	|
| groups                   	| A dict whose keys are Ansible group names and values are a list of  hostnames that are members of the group. Includes all and ungrouped groups:  {"all": […], "web": […], "ungrouped": […]} 	|
| ansible_check_mode       	| A boolean that is true when running in check mode                                                                                                                                           	|
| ansible_play_batch       	| A list of the inventory hostnames that are active in the current batch                                                                                                                      	|
| ansible_play_hosts       	| A list of all of the inventory hostnames that are active in the current  play                                                                                                               	|
| ansible_version          	| A dict with Ansible version info:  {"full": 2.3.1.0", "major": 2, "minor": 3, "revision": 1, "string": "2.3.1.0"}                                                                           	|

• hostvars This is a dictionary that contains all of the variables defined on all of the hosts, keyed by the hostname as known to Ansible. If Ansible has not yet gathered facts on a host, you will not be able to access its facts by using the hostvars variable, unless fact caching is enabled.

  {{ hostvars['db.example.com'].ansible_eth1.ipv4.address }}

• inventory_hostname The inventory_hostname is the hostname of the current host, as known by Ansible.

• groups The groups variable can be useful when you need to access variables for a group of hosts.

I encounter this in playbook vars section:

- name: xxx
  hosts: master
  become: true
  any_errors_fatal: true
  vars:
    NODE_COUNT: "{{ groups['nodes'] | length }}"
  roles:
    - ...

- name: xxx
  any_errors_fatal: true
  shell: "..."
  when: "inventory_hostname == groups.master[0]"

Setting Variables on the Command Line

Variables set by passing -e var=value to ansible-playbook have the highest precedence, which means you can use this to override variables that are already defined.

ansible-playbook greet.yml -e 'greeting="hi there"'

Ansible also allows you to pass a file containing the variables instead of passing them directly on the command line by passing @filename.yml as the argument to -e

ansible-playbook greet.yml -e @greetvars.yml

content in greetvars.yml:

greeting: hiya

Precedence

We’ve covered several ways of defining variables, and it can happen that you define the same variable multiple times for a host, using different values. Avoid this when you can, but if you can’t, then keep in mind Ansible’s precedence rules.

1. (Highest) ansible-playbook -e var=value
2. Task variables
3. Block variables
4. Role and include variables
5. set_fact
6. Registered variables
7. vars_files
8. vars_prompt
9. Play variables
10. Host facts
11. host_vars set on a playbook
12. group_vars set on a playbook
13. host_vars set in the inventory
14. group_vars set in the inventory
15. Inventory variables
16. In defaults/main.yml of a role

Chapter 5. Introducing Mezzanine: Our Test Application

Let’s take a little detour and talk about the differences between running software in development mode on your laptop versus running the software in production. Mezzanine is a great example of an application that is much easier to run in development mode than it is to deploy.

virtualenv venv
source venv/bin/activate
pip install mezzanine
mezzanine-project myproject
cd myproject
sed -i 's/ALLOWED_HOSTS = \[\]/ALLOWED_HOSTS = ["127.0.0.1"]/' settings.py
python manage.py createdb
python manage.py runserver

You’ll be prompted to answer several questions. I answered “yes” to each yes/no question, and accepted the default answer whenever one was available.

Now, let’s look at what happens when you deploy to production.

• PostgreSQL: The Database In production, we want to run a server-based database, because those have better support for multiple, concurrent requests, and server-based databases allow us to run multiple HTTP servers for load balancing. This means we need to deploy a database management system such as MySQL or PostgreSQL (aka Postgres).

  Install the database software.
  Ensure the database service is running.
  Create the database inside the database management system.
  Create a database user who has the appropriate permissions for the database system.
  Configure our Mezzanine application with the database user credentials and 
  connection information.

• Gunicorn: The Application Server

• Nginx: The Web Server

Note: Application server and Web server, their usage is different. Here Nginx is like a reverse proxy for Gunicorn.

• Supervisor: The Process Manager

Chapter 7. Roles: Scaling Up Your Playbooks

Ansible scales down well because simple tasks are easy to implement. It scales up well because it provides mechanisms for decomposing complex jobs into smaller pieces.

In Ansible, the role is the primary mechanism for breaking a playbook into multiple files. This simplifies writing complex playbooks, and it makes them easier to reuse.

Basic Structure of a Role

An Ansible role has a name, such as database. Files associated with the database role go in the roles/database directory, which contains the following files and directories:

| roles/database/tasks/main.yml    	| Tasks                                    	|
| roles/database/files/            	| Holds files to be uploaded to hosts      	|
| roles/database/templates/        	| Holds Jinja2 template files              	|
| roles/database/handlers/main.yml 	| Handlers                                 	|
| roles/database/vars/main.yml     	| Variables that shouldn’t be overridden   	|
| roles/database/defaults/main.yml 	| Default variables that can be overridden 	|
| roles/database/meta/main.yml     	| Dependency information about a role      	|

Each individual file is optional; if your role doesn’t have any handlers, there’s no need to have an empty handlers/main.yml file.

WHERE DOES ANSIBLE LOOK FOR MY ROLES?

Ansible looks for roles in the roles directory alongside your playbooks. It also looks for systemwide roles in /etc/ansible/roles. You can customize the systemwide location of roles by setting the roles_path setting in the defaults section of your ansible.cfg file.

[defaults]
roles_path = ~/ansible_roles

Using Roles in Your Playbooks

- name: deploy mezzanine on vagrant
  ## target hosts
  hosts: web
  vars_files:
    - secrets.yml
  ## role section
  roles:
    - role: database
       ## pass variables into role task
      database_name: "{{ mezzanine_proj_name }}"
      database_user: "{{ mezzanine_proj_name }}"

    - role: mezzanine
      live_hostname: 192.168.33.10.xip.io
      domains:
        - 192.168.33.10.xip.io
        - www.192.168.33.10.xip.io

Note that we can pass in variables when invoking the roles. If these variables have already been defined in the role (either in vars/main.yml or defaults/main.yml), then the values will be overridden with the variables that were passed in.

Pre-Tasks and Post-Tasks

Sometimes you want to run tasks before or after you invoke your roles. Let’s say you want to update the apt cache before you deploy Mezzanine, and you want to send a notification to a Slack channel after you deploy.

Ansible allows you to define a list of tasks that execute before the roles with a pre_tasks section, and a list of tasks that execute after the roles with a post_taskssection.

- name: deploy mezzanine on vagrant
  hosts: web
  vars_files:
    - secrets.yml
  pre_tasks:
    - name: update the apt cache
      apt: update_cache=yes
  roles:
    - role: mezzanine
  post_tasks:
    - name: notify Slack that the servers have been updated
      local_action: >
        slack
        domain=acme.slack.com
        token={{ slack_token }}
        msg="web server {{ inventory_hostname }} configured"

Note: you need to define and put variables in right place, for example, the variables that would be used by multiple roles or playbooks should be put in group_vars/all file.

WHY ARE THERE TWO WAYS TO DEFINE VARIABLES IN ROLES?

When Ansible first introduced support for roles, there was only one place to define role variables, in vars/main.yml. Variables defined in this location have a higher precedence than those defined in the vars section of a play, which meant you couldn’t override the variable unless you explicitly passed it as an argument to the role.

Ansible later introduced the notion of default role variables that go in defaults/main.yml.This type of variable is defined in a role, but has a low precedence, so it will be overridden if another variable with the same name is defined in the playbook.

If you think you might want to change the value of a variable in a role, use a default variable. If you don’t want it to change, use a regular variable.

Some role practices

Note that if for role variables, it’s better to add prefix like <role name>_<var name>. It’s good practice to do this with role variables because Ansible doesn’t have any notion of namespace across roles. This means that variables that are defined in other roles, or elsewhere in a playbook, will be accessible everywhere. This can cause some unexpected behavior if you accidentally use the same variable name in two different roles. For example, for the role called mezzanine, in roles/mezzanine/vars/main.yml file:

mezzanine_user: "{{ ansible_user }}"
mezzanine_venv_home: "{{ ansible_env.HOME }}"
mezzanine_venv_path: "{{ mezzanine_venv_home }}/{{ mezzanine_proj_name }}"
mezzanine_repo_url: git@github.com:lorin/mezzanine-example.git
mezzanine_proj_dirname: project

For role variables in default/main.yml, no need to add prefix because we may intentionally override them elsewhere.

If the role task is very long, you can break it into several task files, for example:

roles/mezzanine/tasks/django.yml
roles/mezzanine/tasks/main.yml
roles/mezzanine/tasks/nginx.yml

In the main.yml task file you can invoke other tasks by using include statement:

- name: install apt packages
  apt: pkg={{ item }} update_cache=yes cache_valid_time=3600
  become: True
  with_items:
    - git
    - supervisor

- include: django.yml
- include: nginx.yml

Note: there’s one important difference between tasks defined in a role and tasks defined in a regular playbook, and that’s when using the copy or template modules.

When invoking copy in a task defined in a role, Ansible will first check the rolename/files/ directory for the location of the file to copy. Similarly, when invoking template in a task defined in a role, Ansible will first check the rolename/templates directory for the location of the template to use.

This means that a task that used to look like this in a playbook:

- name: set the nginx config file
  template: src=templates/nginx.conf.j2 \
  dest=/etc/nginx/sites-available/mezzanine.conf

now looks like this when invoked from inside the role (note the change of the src parameter):

- name: set the nginx config file
  template: src=nginx.conf.j2 dest=/etc/nginx/sites-available/mezzanine.conf
  notify: restart nginx

Creating Role Files and Directories with ansible-galaxy

Ansible ships with another command-line tool we haven’t talked about yet, ansible-galaxy. Its primary purpose is to download roles that have been shared by the Ansible community. But it can also be used to generate scaffolding, an initial set of files and directories involved in a role:

ansible-galaxy init <path>/roles/web

└── roles
    └── web
        ├── README.md
        ├── defaults
        │   └── main.yml
        ├── files
        ├── handlers
        │   └── main.yml
        ├── meta
        │   └── main.yml
        ├── tasks
        │   └── main.yml
        ├── templates
        ├── tests
        │   ├── inventory
        │   └── test.yml
        └── vars
            └── main.yml

Dependent Roles

Ansible supports a feature called dependent roles to deal with this scenario. When you define a role, you can specify that it depends on one or more other roles. Ansible will ensure that roles that are specified as dependencies are executed first.

Let’s say that we create anntp role that configures a host to synchronize its time with an NTP server. Ansible allows us to pass parameters to dependent roles, so let’s also assume that we can pass the NTP server as a parameter to that role.

We specify that the web role depends on the ntp role by creating a roles/web/meta/main.yml file and listing ntp as a role, with a parameter:

dependencies:
    - { role: ntp, ntp_server=ntp.ubuntu.com }

Ansible Galaxy

Whether you want to reuse a role somebody has already written, or you just want to see how someone else solved the problem you’re working on, Ansible Galaxy can help you out. Ansible Galaxy is an open source repository of Ansible roles contributed by the Ansible community. The roles themselves are stored on GitHub.

Chapter 8. Complex Playbooks

This chapter touches on those additional features, which makes it a bit of a grab bag.

Dealing with Badly Behaved Commands

What if we didn’t have a module that could invoke equivalent commands (wasn’t idempotent)? The answer is to use changed_when and failed_when clauses to change how Ansible identifies that a task has changed state or failed.

First, we need to understand the output of this command the first time it’s run, and the output when it’s run the second time.

- name: initialize the database
  django_manage:
    command: createdb --noinput --nodata
    app_path: "{{ proj_path }}"
    virtualenv: "{{ venv_path }}"
  failed_when: False
  register: result

- debug: var=result

- fail:

failed_when: False is to close task fail, so ansible play will continue to execute. We can run several times of the playbook and see different register variable output. fail statement here is to stop the execution.

Some module may not report changed state even though it did make change in target machine, so we can check if state changed ourselves by using changed_when clause:

- name: initialize the database
  django_manage:
    command: createdb --noinput --nodata
    app_path: "{{ proj_path }}"
    virtualenv: "{{ venv_path }}"
  register: result
  changed_when: '"Creating tables" in result.out|default("")'

We use filter here in changed_when since register variable sometimes doesn’t have out field. Alternatively, we could provide a default value for result.out if it doesn’t exist by using the Jinja2 default filter.

Filter

Filters are a feature of the Jinja2 templating engine. Since Ansible uses Jinja2 for evaluating variables, as well as for templates, you can use filters inside  in your playbooks, as well as inside your template files. Using filters resembles using Unix pipes, whereby a variable is piped through a filter. Jinja2 ships with a set of built-in filters. In addition, Ansible ships with its own filters to augment the Jinja2 filters.

The Default Filter

"HOST": "{{ database_host | default('localhost') }}"

If the variable database_host is defined, the braces will evaluate to the value of that variable. If the variable database_host is not defined, the braces will evaluate to the string localhost.

Filters for Registered Variables

Let’s say we want to run a task and print out its output, even if the task fails. However, if the task does fail, we want Ansible to fail for that host after printing the output.

- name: Run myprog
  command: /opt/myprog
  register: result
  ignore_errors: True

- debug: var=result

- debug: msg="Stop running the playbook if myprog failed"
  failed_when: result|failed

a list of filters you can use on registered variables to check the status:

| Name    	| Description                                           	|
|---------	|-------------------------------------------------------	|
| failed  	| True if a registered value is a task that failed      	|
| changed 	| True if a registered value is a task that changed     	|
| success 	| True if a registered value is a task that succeeded   	|
| skipped 	| True if a registered value is a task that was skipped 	|

The basename filter will let us extract the index.html part of the filename from the full path, allowing us to write the playbook without repeating the filename:

vars:
    homepage: /usr/share/nginx/html/index.html
  tasks:
  - name: copy home page
    copy: src=files/{{ homepage | basename }} dest={{ homepage }}

Lookups

Sometimes a piece of configuration data you need lives somewhere else. Maybe it’s in a text file or a .csv file, and you don’t want to just copy the data into an Ansible variable file because now you have to maintain two copies of the same data.

Ansible has a feature called lookups that allows you to read in configuration data from various sources and then use that data in your playbooks and template.

| Name     	| Description                        	|
|----------	|------------------------------------	|
| file     	| Contents of a file                 	|
| password 	| Randomly generate a password       	|
| pipe     	| Output of locally executed command 	|
| env      	| Environment variable               	|
| template 	| Jinja2 template after evaluation   	|
| csvfile  	| Entry in a .csv file               	|
| dnstxt   	| DNS TXT record                     	|
| redis_kv 	| Redis key lookup                   	|
| etcd     	| etcd key lookup                    	|

You can invoke lookups in your playbooks between , or you can put them in templates.

Note all Ansible lookup plugins execute on the control machine, not the remote host.

file

Let’s say you have a text file on your control machine that contains a public SSH key that you want to copy to a remote server.

- name: Add my public key as an EC2 key
  ec2_key: name=mykey key_material="{{ lookup('file',  '/Users/lorin/.ssh/id_rsa.pub') }}"

pipe

The pipe lookup invokes an external program on the control machine and evaluates to the program’s output on standard out.

For example, if our playbooks are version controlled using git, and we want to get the SHA-1 value of the most recent git commit, we could use the pipe lookup

- name: get SHA of most recent commit
  debug: msg="{{ lookup('pipe', 'git rev-parse HEAD') }}"

TASK: [get the sha of the current commit] *************************************
ok: [myserver] => {
    "msg": "e7748af0f040d58d61de1917980a210df419eae9"
}

env

The env lookup retrieves the value of an environment variable set on the control machine.For example, we could use the lookup like this:

- name: get the current shell
  debug: msg="{{ lookup('env', 'SHELL') }}"

TASK: [get the current shell] *************************************************
ok: [myserver] => {
    "msg": "/bin/zsh"
}

password

The password lookup evaluates to a random password, and it will also write the password to a file specified in the argument. For example, if we want to create a Postgres user named deploy with a random password and write that password to deploy-password.txton the control machine, we can do this:

- name: create deploy postgres user
  postgresql_user:
    name: deploy
    password: "{{ lookup('password', 'deploy-password.txt') }}"

template

The template lookup lets you specify a Jinja2 template file, and then returns the result of evaluating the template.

csvfile

The csvfile lookup reads an entry from a .csv file. For example, we have a users.csv file:

username,email
lorin,lorin@ansiblebook.com
john,john@example.com
sue,sue@example.org

lookup('csvfile', 'sue file=users.csv delimiter=, col=1')

In the case of csvfile, the first argument is an entry that must appear exactly once in column 0 (the first column, 0-indexed) of the table.

In our example, we want to look in the file named users.csv and locate where the fields are delimited by commas, look up the row where the value in the first column is sue, and return the value in the second column (column 1, indexed by 0). This evaluates to sue@example.org.

etcd

Etcd is a distributed key-value store, commonly used for keeping configuration data and for implementing service discovery. You can use the etcd lookup to retrieve the value of a key.

For example, let’s say that we have an etcd server running on our control machine, and we set the key weather to the value cloudy by doing something like this:

curl -L http://127.0.0.1:4001/v2/keys/weather -XPUT -d value=cloudy

- name: look up value in etcd
  debug: msg="{{ lookup('etcd', 'weather') }}"

TASK: [look up value in etcd] *************************************************
ok: [localhost] => {
    "msg": "cloudy"
}

By default, the etcd lookup looks for the etcd server at http://127.0.0.1:4001, but you can change this by setting the ANSIBLE_ETCD_URL environment variable before invoking ansible-playbook.

More Complicated Loops

Up until this point, whenever we’ve written a task that iterates over a list of items, we’ve used the with_items clause to specify a list of items. Although this is the most common way to do loops, Ansible supports other mechanisms for iteration.

| Name                     	| Input                	| Looping strategy                  	|
|--------------------------	|----------------------	|-----------------------------------	|
| with_items               	| List                 	| Loop over list elements           	|
| with_lines               	| Command to execute   	| Loop over lines in command output 	|
| with_fileglob            	| Glob                 	| Loop over filenames               	|
| with_first_found         	| List of paths        	| First file in input that exists   	|
| with_dict                	| Dictionary           	| Loop over dictionary elements     	|
| with_flattened           	| List of lists        	| Loop over flattened list          	|
| with_indexed_items       	| List                 	| Single iteration                  	|
| with_nested              	| List                 	| Nested loop                       	|
| with_random_choice       	| List                 	| Single iteration                  	|
| with_sequence            	| Sequence of integers 	| Loop over sequence                	|
| with_subelements         	| List of dictionaries 	| Nested loop                       	|
| with_together            	| List of lists        	| Loop over zipped list             	|
| with_inventory_hostnames 	| Host pattern         	| Loop over matching hosts          	|

The official documentation covers these quite thoroughly, so I’ll show examples from just a few of them to give you a sense of how they work.

with_lines

The with_lines looping construct lets you run an arbitrary command on your control machine and iterate over the output, one line at a time. For example, read a file and iterate over its contents line by line.

- name: Send out a slack message
  slack:
    domain: example.slack.com
    token: "{{ slack_token }}"
    msg: "{{ item }} was in the list"
  with_lines:
    - cat files/turing.txt

with_fileglob

The with_fileglob construct is useful for iterating over a set of files on the control machine.

For example, iterate over files that end in .pub in the /var/keys directory, as well as a keys directory next to your playbook. It then uses the file lookup plugin to extract the contents of the file, which are passed to the authorized_key module.

- name: add public keys to account
  authorized_key: user=deploy key="{{ lookup('file', item) }}"
  with_fileglob:
    - /var/keys/*.pub
    - keys/*.pub

with_dict

The with_dict construct lets you iterate over a dictionary instead of a list. When you use this looping construct, the item loop variable is a dictionary with two fields:

- name: iterate over ansible_eth0
   debug: msg={{ item.key }}={{ item.value }}
   with_dict: "{{ ansible_eth0.ipv4 }}"

Looping Constructs as Lookup Plugins

Ansible implements looping constructs as lookup plugins. That means you can alter the form of lookup to perform as a loop:

- name: Add my public key as an EC2 key
  ec2_key: name=mykey key_material="{{ item }}"
  with_file: /Users/lorin/.ssh/id_rsa.pub

Here we prefix with_ with file lookup plugin. Typically, you use a lookup plugin as a looping construct only if it returns a list,

Loop Controls

With version 2.1, Ansible provides users with more control over loop handling.

Setting the Variable Name

The loop_var control allows us to give the iteration variable a different name than the default name, item:

- user:
    name: "{{ user.name }}"
  with_items:
  ## list of dict
    - { name: gil }
    - { name: sarina }
    - { name: leanne }
  loop_control:
    loop_var: user

Next one is a advanced usage, use include with with_items, we loop over multiple task at once, in current task we include a task called vhosts.yml which will be executed 3 times with different parameters passed in:

- name: run a set of tasks in one loop
  include: vhosts.yml
  with_items:
    - { domain: www1.example.com }
    - { domain: www2.example.com }
    - { domain: www3.example.com }
  loop_control:
    loop_var: vhost

The vhosts.yml file that is going to be included may also contain with_items in some tasks. This would produce a conflict, as the default loop_var item is used for both loops at the same time.

To prevent a naming collision, we specify a different name for loop_var in the outer loop.

- name: create nginx directories
  file:
    path: /var/www/html/{{ vhost.domain }}/{{ item }}
  state: directory
  with_items:
    - logs
    - public_http
    - public_https
    - includes

- name: create nginx vhost config
  template:
    src: "{{ vhost.domain }}.j2"
    dest: /etc/nginx/conf.d/{{ vhost.domain }}.conf

Labeling the Output

The label control was added in Ansible 2.2 and provides some control over how the loop output will be shown to the user during execution.

- name: create nginx vhost configs
  template:
    src: "{{ item.domain }}.conf.j2"
    dest: "/etc/nginx/conf.d/{{ item.domain }}.conf"
  with_items:
    - { domain: www1.example.com, ssl_enabled: yes }
    - { domain: www2.example.com }
    - { domain: www3.example.com,
      aliases: [ edge2.www.example.com, eu.www.example.com ] }
  loop_control:
    label: "for domain {{ item.domain }}"

TASK [create nginx vhost configs] **********************************************
ok: [localhost] => (item=for domain www1.example.com)
ok: [localhost] => (item=for domain www2.example.com)
ok: [localhost] => (item=for domain www3.example.com)

Includes

The include feature allows you to include tasks or even whole playbooks, depending on where you define an include. It is often used in roles to separate or even group tasks and task arguments to each task in the included file.

For example, you can extract different part of tasks, put them into a separate yml file and include it into another task along with common arguments:

# nginx_include.yml file
- name: install nginx
  package:
    name: nginx

- name: ensure nginx is running
  service:
    name: nginx
    state: started
    enabled: yes

- include: nginx_include.yml
  tags: nginx
  become: yes
  when: ansible_os_family == 'RedHat'

Ansible Tags: If you have a large playbook, it may become useful to be able to run only a specific part of it rather than running everything in the playbook. Ansible supports a tags: attribute for this reason.

Dynamic includes

A common pattern in roles is to define tasks specific to a particular operating system into separate task files.

- include: Redhat.yml
  when: ansible_os_family == 'Redhat'

- include: Debian.yml
  when: ansible_os_family == 'Debian'

Since version 2.0, Ansible allows us to dynamically include a file by using variable substitution:

- include: "{{ ansible_os_family }}.yml"
  static: no

However, there is a drawback to using dynamic includes: ansible-playbook --list-tasks might not list the tasks from a dynamic include if Ansible does not have enough information to populate the variables that determine which file will be included.

You can use ansible-playbook <playbook> --list-tasks to list all the tasks in it.

Role includes

A special include is the include_role clause. In contrast with the role clause, which will use all parts of the role, the include_role not only allows us to selectively choose what parts of a role will be included and used, but also where in the play.

- name: install php
  include_role:
    name: php

This will include and run main.yml from the php role, remember a role can have multiple tasks yml files: main.yml and others.

- name: install php
  include_role:
    name: php
    tasks_from: install

This will include and run install.yml from php role.

Blocks

Much like the include clause, the block clause provides a mechanism for grouping tasks. The block clause allows you to set conditions or arguments for all tasks within a block at once:

- block:
  - name: install nginx
    package:
      name: nginx
  - name: ensure nginx is running
    service:
      name: nginx
      state: started
      enabled: yes
  become: yes
  when: "ansible_os_family == 'RedHat'"

The become and when apply for both tasks.

Error Handling with Blocks

Dealing with error scenarios has always been a challenge. Historically, Ansible has been error agnostic in the sense that errors and failures may occur on a host. Ansible’s default error-handling behavior is to take a host out of the play if a task fails and continue as long as there are hosts remaining that haven’t encountered errors.

- block:
  - debug: msg="You will see a failed tasks right after this"
  - command: /bin/false
  - debug: "You won't see this message"
  rescue: # Tasks to be executed in case of a failure in block clause
  - debug: "You only see this message in case of an failure in the block"
  always: # Tasks to always be executed
  - debug: "This will be always executed"

If you have some programming experience, the way error handling is implemented may remind you of the try-catch-finally paradigm, and it works much the same way.

Encrypting Sensitive Data with Vault

Ansible provides an alternative solution: instead of keeping the secrets.yml file out of version control, we can commit an encrypted version. That way, even if our version-control repository were compromised, the attacker would not have access to the contents of the secrets.yml file unless he also had the password used for the encryption.

The ansible-vault command-line tool allows you to create and edit an encrypted file that ansible-playbook will recognize and decrypt automatically, given the password.

ansible-vault create secrets.yml
ansible-vault encrypt secrets.yml

You will be prompted for a password, and then ansible-vault will launch a text editor so that you can populate the file. It launches the editor specified in the $EDITOR environment variable. If that variable is not defined, it defaults to vim.

ansible-playbook <playbook> --ask-vault-pass
ansible-playbook <playbook> --vault-password-file ~/password.txt

If the argument to --vault-password-file has the executable bit set, Ansible will execute it and use the contents of standard out as the vault password. This allows you to use a script to provide the password to Ansible.

| Command                        	| Description                                       	|
|--------------------------------	|---------------------------------------------------	|
| ansible-vault encrypt file.yml 	| Encrypt the plain-text file.yml file              	|
| ansible-vault decrypt file.yml 	| Decrypt the encrypted file.yml file               	|
| ansible-vault view file.yml    	| Print the contents of the encrypted file.yml file 	|
| ansible-vault create file.yml  	| Create a new encrypted file.yml file              	|
| ansible-vault edit file.yml    	| Edit an encrypted file.yml file                   	|
| ansible-vault rekey file.yml   	| Change the password on an encrypted file.yml file 	|

Chapter 9. Customizing Hosts, Runs, and Handlers

In this chapter, we cover Ansible features that provide customization by controlling which hosts to run against, how tasks are run, and how handlers are run.

Patterns for Specifying Hosts

Instead of specifying a single host or group for a play, you can specify a pattern. You’ve already seen the all pattern, which will run a play against all known hosts:

hosts: all

You can specify a union of two groups with a colon. You specify all dev and staging machines as follows:

hosts: dev:staging

| Action                    	| Example Usage               	|
|---------------------------	|-----------------------------	|
| All hosts                 	| all                         	|
| All hosts                 	| *                           	|
| Union                     	| dev:staging                 	|
| Intersection              	| dev:&database               	|
| Exclusion                 	| dev:!queue                  	|
| Wildcard                  	| *.example.com               	|
| Range of numbered servers 	| web[5:12]                   	|
| Regular expression        	| ~web\d+\.example\.(com|org) 	|

Ansible supports multiple combinations of patterns—for example:

hosts: dev:staging:&database:!queue

Limiting Which Hosts Run

Use the -l hosts or --limit hosts flag to tell Ansible to limit the hosts to run the playbook against the specified list of hosts

ansible-playbook -l hosts playbook.yml
ansible-playbook --limit hosts playbook.yml
ansible-playbook -l 'staging:&database' playbook.yml

Running a Task on the Control Machine

Sometimes you want to run a particular task on the control machine instead of on the remote host. Ansible provides the local_action clause for tasks to support this. For example, when we check the node ready status in k8s cluster.

Imagine that the server we want to install Mezzanine onto has just booted, so that if we run our playbook too soon, it will error out because the server hasn’t fully started up yet. We could start off our playbook by invoking the wait_for module to wait until the SSH server is ready to accept connections before we execute the rest of the playbook.

- name: wait for ssh server to be running
  local_action: wait_for port=22 host="{{ inventory_hostname }}" search_regex=OpenSSH

Note that inventory_hostname evaluates to the name of the remote host, not localhost. That’s because the scope of these variables is still the remote host, even though the task is executing locally.

If your play involves multiple hosts, and you use local_action, the task will be executed multiple times, one for each host. You can restrict this by using run_once

Running a Task on a Machine Other Than the Host

Sometimes you want to run a task that’s associated with a host, but you want to execute the task on a different server. You can use the delegate_to clause to run the task on a different host.

- name: enable alerts for web servers
  hosts: web
  tasks:
    - name: enable alerts
      nagios: action=enable_alerts service=web host={{ inventory_hostname }}
      delegate_to: nagios.example.com

In this example, Ansible would execute the nagios task on nagios.example.com, but the inventory_hostname variable referenced in the play would evaluate to the web host.

Note: if you specify delegate_to: localhost to control machine, it’s the same as local_action, also the same as connection: local

Running on One Host at a Time

By default, Ansible runs each task in parallel across all hosts. Sometimes you want to run your task on one host at a time. The canonical example is when upgrading application servers that are behind a load balancer. Typically, you take the application server out of the load balancer, upgrade it, and put it back. But you don’t want to take all of your application servers out of the load balancer, or your service will become unavailable.

You can use the serial clause on a play to tell Ansible to restrict the number of hosts that a play runs on.

- name: upgrade packages on servers behind load balancer
  hosts: myhosts
  serial: 1
  tasks:
    - name: get the ec2 instance id and elastic load balancer id
      ec2_facts:

    - name: take the host out of the elastic load balancer
      local_action: ec2_elb
      args:
        instance_id: "{{ ansible_ec2_instance_id }}"
        state: absent

    - name: upgrade packages
      apt: update_cache=yes upgrade=yes

    - name: put the host back in the elastic load balancer
      local_action: ec2_elb
      args:
        instance_id: "{{ ansible_ec2_instance_id }}"
        state: present
        ec2_elbs: "{{ item }}"
      with_items: ec2_elbs

In our example, we pass 1 as the argument to the serial clause, telling Ansible to run on only one host at a time. If we had passed 2, Ansible would have run two hosts at a time.

Normally, when a task fails, Ansible stops running tasks against the host that fails, but continues to run against other hosts. In the load-balancing scenario, you might want Ansible to fail the entire play before all hosts have failed a task. Otherwise, you might end up with the situation where you have taken each host out of the load balancer, and have it fail, leaving no hosts left inside your load balancer.

You can use a max_fail_percentage clause along with the serial clause to specify the maximum percentage of failed hosts before Ansible fails the entire play. For example, assume that we specify a maximum fail percentage of 25%, as shown here:

- name: upgrade packages on servers behind load balancer
  hosts: myhosts
  serial: 1
  max_fail_percentage: 25
  tasks:
    # tasks go here

If you want Ansible to fail if any of the hosts fail a task, set the max_fail_percentage to 0.

Note: any_errors_fatal: true is just like set max_fail_percentage to 0, with the any_errors_fatal option, any failure on any host in a multi-host play will be treated as fatal and Ansible will exit immediately without waiting for the other hosts.

We can get even more sophisticated. For example, you might want to run the play on one host first, to verify that the play works as expected, and then run the play on a larger number of hosts in subsequent runs.

- name: configure CDN servers
  hosts: cdn
  serial:
    - 1
    - 30%
  tasks:
    # tasks go here

In the preceding play with 30 CDN hosts, on the first batch run Ansible would run against one host, and on each subsequent batch run it would run against at most 30% of the hosts (e.g., 1, 10, 10, 9).

Running Only Once

Using run_once can be particularly useful when using local_action if your playbook involves multiple hosts, and you want to run the local task only once:

- name: run the task locally, only once
  local_action: command /opt/my-custom-command
  run_once: true

Running Strategies

The strategy clause on a play level gives you additional control over how Ansible behaves per task for all hosts.

The default behavior we are already familiar with is the linear strategy. This is the strategy in which Ansible executes one task on all hosts and waits until the task has completed (of failed) on all hosts before it executes the next task on all hosts. As a result, a task takes as much time as the slowest host takes to complete the task.

Linear

Note: I forget that host file can define variable, here sleep_seconds can be referred in task:

one   sleep_seconds=1
two   sleep_seconds=6
three  sleep_seconds=10

Note that the orders show up is the complete order in target host, first done at top.

TASK [setup] *******************************************************************
ok: [two]
ok: [three]
ok: [one]

Free

Another strategy available in Ansible is the free strategy. In contrast to linear, Ansible will not wait for results of the task to execute on all hosts. Instead, if a host completes one task, Ansible will execute the next task on that host.

- hosts: all
  strategy: free
  tasks:
     ...

Advanced Handlers

When we covered handlers, you learned that they are usually executed after all tasks, once, and only when they get notified. But keep in mind there are not only tasks, but pre_tasks, tasks, and post_tasks.

Each tasks section in a playbook is handled separately; any handler notified in pre_tasks, tasks, or post_tasks is executed at the end of each section. As a result, it is possible to execute one handler several times in one play:

Note: the rest of Chapter 9 is useless for me now, just skip it.

Chapter 16. Debugging Ansible Playbooks

Humane Error Messages

Enable the plugin by adding the following to the defaults section of ansible.cfg:

[defaults]
stdout_callback = debug

the debug callback plugin makes this output much easier for a human to read, for example the format is like this:

TASK [check out the repository on the host] *************************************
fatal: [web]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git clone --origin origin '' /home/vagrant/mezzanine/mezzani
...
}

STDERR:

Cloning into '/home/vagrant/mezzanine/mezzanine_example'...
Permission denied (publickey).
fatal: Could not read from remote repository.
...
MSG:

Cloning into '/home/vagrant/mezzanine/mezzanine_example'...
Permission denied (publickey).
fatal: Could not read from remote repository.
...

Debugging SSH Issues

Sometimes Ansible fails to make a successful SSH connection with the host. When this happens, it’s helpful to see exactly what arguments Ansible is passing to the underlying SSH client so you can reproduce the problem manually on the command line.

If you invoke ansible-playbook with the -vvv argument, you can see the exact SSH commands that Ansible invokes. This can be handy for debugging.

Note that usually I use -v flag

Sometimes you might need to use -vvvv when debugging a connection issue, in order to see an error message that the SSH client is throwing.

The Debug Module

We’ve used the debug module several times in this book. It’s Ansible’s version of a print statement.

- debug: var=myvariable
- debug: msg="The value of myvariable is {{ var }}"
- debug: var=hostvars[inventory_hostname]

Playbook Debugger

Ansible 2.1 added support for an interactive debugger. To enable debugging, add strategy: debug to your play; for example:

- name: an example play
  strategy: debug
  tasks:
    ...

If debugging is enabled, Ansible drops into the debugger when a task fails, for example, I write a task like this:

- name: install xxx package
  yum:
    name: xxx
    state: latest

TASK [install.components : interactive debug] ************************************************************************
fatal: [myk8s2.fyre.ibm.com]: FAILED! => {
    "changed": false,
    "rc": 126,
    "results": [
        "No package matching 'xxx' found available, installed or updated"
    ]
}

MSG:

No package matching 'xxx' found available, installed or updated

Debugger invoked
(debug) p task

Let’s see the command list

| Command              | Description                                 |
|----------------------|---------------------------------------------|
| p var                | Print out the value of a supported variable |
| task.args[key]=value | Modify an argument for the failed task      |
| vars[key]=value      | Modify the value of a variable              |
| r                    | rerun the failed task                       |
| c                    | continue execute next                       |
| q                    | abort the play                              |
| help                 | show help message                           |

variables supported by the debugger

| Command     | Description                            |
|-------------|----------------------------------------|
| p task      | the name of the failed task            |
| p task.args | The module arguments                   |
| p result    | The result returned by the failed task |
| p vars      | Value of all known variables           |
| p vars[key] | Value of one variable                  |

The Assert Module

The assert module will fail with an error if a specified condition is not met. For example, to fail the playbook if there’s no eth1 interface:

- name: assert that eth1 interface exists
  assert:
    that: ansible_eth1 is defined

When debugging a playbook, it can be helpful to insert assertions so that a failure happens as soon as any assumption you’ve made has been violated.

Keep in mind that the code in an assert statement is Jinja2, not Python.

Checking Your Playbook Before Execution

The ansible-playbook command supports several flags that allow you to sanity check your playbook before you execute it.

Syntax Check

The --syntax-check flag checks that your playbook’s syntax is valid, but it does not execute it.

ansible-playbook --syntax-check -i <host file> playbook.yml

List Hosts

The --list-hosts flag outputs the hosts that the playbook will run against, but it does not execute the playbook.

ansible-playbook --list-hosts -i <host file> playbook.yml

List Tasks

Outputs the tasks that the playbook will run against. It does not execute the playbook.

ansible-playbook --list-tasks -i <host file> playbook.yml

Check Mode

The -C and --check flags run Ansible in check mode (sometimes known as dry-run), which tells you whether each task in the playbook will modify the host, but does not make any changes to the server.

ansible-playbook --check -i <host file> playbook.yml

One of the challenges with using check mode is that later parts of a playbook might succeed only if earlier parts of the playbook were executed.

Diff (Show File Changes)

The -D and -diff flags output differences for any files that are changed on the remote machine. It’s a helpful option to use in conjunction with --check to show how Ansible would change the file if it were run normally:

ansible-playbook --diff --check -i <host file> playbook.yml

If Ansible would modify any files (e.g., using modules such as copy, template, and lineinfile), it will show the changes in .diff format, like this:

TASK: [set the gunicorn config file] ******************************************
--- before: /home/vagrant/mezzanine-example/project/gunicorn.conf.py
+++ after: /Users/lorin/dev/ansiblebook/ch06/playbooks/templates/gunicor
n.conf.py.j2
@@ -1,7 +1,7 @@
 from __future__ import unicode_literals
 import multiprocessing

 bind = "127.0.0.1:8000"
 workers = multiprocessing.cpu_count() * 2 + 1
-loglevel = "error"
+loglevel = "warning"
 proc_name = "mezzanine-example"

Limiting Which Tasks Run

Sometimes you don’t want Ansible to run every single task in your playbook, particularly when you’re first writing and debugging the playbook. Ansible provides several command-line options that let you control which tasks run.

Step

The --step flag, shown in Example 16-7, has Ansible prompt you before running each task, like this:

Perform task: install packages (y/n/c):

You can choose to execute the task (y), skip it (n), or tell Ansible to continue running the rest of the playbook without prompting you ©.

ansible-playbook -i <host file> --step playbook.yml

Start-at-Task

The --start-at-task taskname flag tells Ansible to start running the playbook at the specified task, instead of at the beginning. This can be handy if one of your tasks failed because there was a bug in one of your tasks, and you want to rerun your playbook starting at the task you just fixed.

ansible-playbook -i <host file> --start-at-task="install packages" playbook.yml

Tags

Ansible allows you to add one or more tags to a task or a play. For example, here’s a play that’s tagged with foo and a task that’s tagged with bar and quux:

- hosts: myservers
  tags:
   - foo
  tasks:
   - name: install editors
     apt: name={{ item }}
     with_items:
       - vim
       - emacs
       - nano

   - name: run arbitrary command
     command: /opt/myprog
     tags:
       - bar
       - quux

Use the -t tagnames or --tags tagnames flag to tell Ansible to run only plays and tasks that have certain tags. Use the --skip-tags tagnames flag to tell Ansible to skip plays and tasks that have certain tags.

ansible-playbook -t foo,bar playbook.yml
ansible-playbook --tags=foo,bar playbook.yml
ansible-playbook --skip-tags=baz,quux playbook.yml