Ansible Static Dynamic Inventory

At TaskRabbit we use Ansible to configure and manage our servers. Ansible is a great tool which allows you write easy-to-use playbooks to configure your servers, deploy your applications, and more.

The problem

Normally, you run ansible commands from your laptop as you need them. This is great when provisioning or deploying, but annoying that it would be hard to automate. Ansible has a product called Ansible Tower which allows you to run those same commands via a web-UI, schedule them, and respond to web hooks. Tower is a nifty piece of software that does a lot of things right, however we were having trouble keeping out inventories (lists of servers) up-to-date between the lists in our ansible git repository and the the Tower server itself.

The main issue is a change in philosophy. Ansible (the CLI tool) expects that your inventories live local to the project in MAKEFILE-like files located in sensible places like ./inventories/production and ./inventories/staging. Ansible Tower expects that your inventory is dynamic, and always obtainable from a remote source like Amazon EC2’s API, or from a VMware Cluster. While we do use these services to host our servers, not all servers that are present should be ansible’d, and more importantly, not all variables that ansible needs will be obtainable from those sources.

In the Ansible project repo, you can keep both the groups and lists of servers, along with variables like this:

1###########
2## HOSTS ##
3###########
4
5mysql-master.myapp.com
6mysql-slave1.myapp.com
7mysql-slave2.myapp.com
8
9redis.myapp.com
10
11web1.myapp.com
12web2.myapp.com
13web3.myapp.com
14
15resque1.myapp.com
16resque2.myapp.com
17
18############
19## GROUPS ##
20############
21
22[production]
23mysql-master.myapp.com
24mysql-slave1.myapp.com
25mysql-slave2.myapp.com
26redis.myapp.com
27web1.myapp.com
28web2.myapp.com
29web3.myapp.com
30resque1.myapp.com
31resque2.myapp.com
32
33[production:vars]
34host_memory=8GB
35host_disk=20GB
36ansible_ssh_user=root
37
38## DB ##
39
40[mysql]
41mysql-master.myapp.com
42mysql-slave1.myapp.com
43mysql-slave2.myapp.com
44
45[mysql:master]
46mysql-master.myapp.com
47
48[mysql:vars]
49host_memory=32GB
50host_disk=5120GB
51
52[redis]
53redis.myapp.com
54
55[app]
56web1.myapp.com
57web2.myapp.com
58web3.myapp.com
59resque1.myapp.com
60resque2.myapp.com
61
62[app:unicorn]
63web1.myapp.com
64web2.myapp.com
65web3.myapp.com
66
67[app:resque]
68resque1.myapp.com
69resque2.myapp.com
70

This type of layout allows you to define things in a simple way: — hosts belong to groups — groups can have variables — you can override default variables with later group definitions down the file.

To demonstrate this, you can see how all servers start with 8GB of RAM, but the mysql group later overrides this to 32GB. You also get the added bonus of having your entire infrastructure defined in one place.

Our workflow appends this file when we add and remove servers. This means that with a simple git pull you can be sure that any ansible command you run will be run on the correct collection of servers. We wanted Tower to source the same file developers would be using locally, and not reading in (potentially divergent information) via APIs.

Ansible Tower has a feature called "Dynamic Inventory" which allows you to define your inventory via some other method, as long as it presents a standardized JSON output. Tower can reference these things as what they call an "Inventory Script". Using these tools, the question became: "How can we source a file as if it were a changing API?"

The answer had a few parts (in ruby):

1. Find the inventory file

Tower does not keep the git repo of your ansible project(s) in a single place. It versions them and moves them around as you update it. To that end, finding the most current version of your ./inventories/produciton file is non trivial:

1class InventoryFinder
2
3  def find(inventory_file)
4    # On Production server
5    if File.exists? '/var/lib/awx/projects/'
6      folder = Dir.glob('/var/lib/awx/projects/*').max { |a,b| File.ctime(a) <=> File.ctime(b) }
7      return folder + '/inventories/' + inventory_file
8    # Assume we are within the proper project
9    else
10      return File.dirname(__FILE__) + '/../inventories/' + inventory_file
11    end
12  end
13
14end

2. Parse the Inventory

You can define groups and variables in a few legal ways within an inventory file. You can do the [group:vars] method in the example above, or you can do it in-line as you define the server for the first time. Keeping all this in mind, here’s our parser:

1class InventoryParser
2
3  def initialize(inventory_path)
4    @inventory_path = inventory_path
5    @data = {
6      "_meta" => {
7        "hostvars" => {}
8      }
9    }
10  end
11
12  def inventory_path
13    @inventory_path
14  end
15
16  def data
17    @data
18  end
19
20  def ignored_variables
21    [
22      'ansible_ssh_user'
23    ]
24  end
25
26  def file_lines
27    File.read( inventory_path ).split("\n")
28  end
29
30  def parse
31    current_section = nil
32
33    file_lines.each do |line|
34      parts = line.split(' ')
35      next if parts.length == 0
36      next if parts.first[0] == "#"
37      next if parts.first[0] == "/"
38      if parts.first[0] == '['
39        current_section = parts.first.gsub('[','').gsub(']','')
40        if data[current_section].nil? && !current_section.include?(':vars')
41          data[current_section] = []
42        end
43        next
44      end
45
46      # varaible block
47      if !current_section.nil? && current_section.include?(':vars')
48        parts = line.split('=')
49        key   = parts[0]
50        value = parts[1]
51        col   = current_section.split(':')
52        col.pop
53        group = col.join(':')
54        fill_hosts_with_group_var(group, key, value)
55      # host block (could still have in-line variables)
56      else
57        hostname = parts.shift
58        ensure_host_variables(hostname)
59        d = {}
60
61        while parts.length > 0
62          part = parts.shift
63          words = part.split('=')
64          d[words.first] = words.last unless ignored_variables.include? words.first
65        end
66
67        data[current_section].push(hostname) if current_section
68        d.each do |k,v|
69          data["_meta"]["hostvars"][hostname][k] = v
70        end
71
72      end
73    end
74
75    return data
76  end
77
78  def ensure_host_variables(hostname)
79    if data["_meta"]["hostvars"][hostname].nil?
80      data["_meta"]["hostvars"][hostname] = {}
81    end
82  end
83
84  def fill_hosts_with_group_var(group, key, value)
85    return if ignored_variables.include? key
86
87    if value.include?("'") || value.include?('"')
88      value = eval(value)
89    end
90
91    data[group].each do |hostname|
92      ensure_host_variables(hostname)
93      data["_meta"]["hostvars"][hostname][key] = value
94    end
95  end
96
97end

You will also note that we choose to ignore certain variables, via ignored_variables, that we want defined somewhere else within ansible tower (for example SSH options).

As a note, one feature of ansible’s inventory DSL that is not supported here is the notion of children

3. Running it

Once those classes are defined, you can create a single file (per environment) like so:

1#!/usr/bin/env ruby
2
3require 'json'
4
5class InventoryFinder
6 #...
7end
8
9class InventoryParser
10 #...
11end
12
13path = InventoryFinder.new.find('production')
14data = InventoryParser.new(path).parse
15
16puts JSON.pretty_generate( data )

You can load this code into the dynamic inventory and it will be ready to run!

4. Keeping it in sync

The final step is to ensure that any time a job is run from Tower, both the project repository and inventory are always updated. There are a few hooks you need to enable to do so:

First, on the setting for the project, you can enable a git pull before each project run. Be sure to enable Update on Launch under SCM options.

Then, the same option, Update on Launch can be enabled under the inventory source. When you define your inventory, you need to source is a "custom script", and from there, you can choose the inventory reader defined above.

With this place, we are able to have our cake and eat it too: — one file which contains all of our configuration — allow developers to keep an up-to-date inventory source locally within the git ansible project — Ansible Tower can source that file, and ensure that it is up-to-date before we run any job