Rebuilding Capistrano with Ansible

Introduction

At TaskRabbit we use Ansible to configure and manage our servers. Ansible is a great tool which allows you write easy-to-use playbooks to configure your servers, deploy your applications, and more. The "More" part was what led us to switch from Chef to Ansbile. While both tools can have a "provision" action, you can make playbooks for all sorts of things with Ansible, including application deployment!

For the past 4 years, TaskRabbit was using Capistrano to deploy our Rails applications. We built a robust and feature-rich set of plugins which:

• changed the way we logged during deployments (better timestamps, shared the deployment logs with the rest of the team, etc)
• became a rails module we could plug into each of our applications with minimal configuration changes
• standardized some of our deployment steps
• and codified our best practices (ie: cap staging deploy:migrations should work for all apps; All apps should wait for the Unicorns to reboot before clearing cache, etc)

Eventually, we started adding more and more non-rails (Sinatra), and then non-ruby (Node.js) apps. I’ve written before on how you can use Capistrano to deploy anything, including node.js applications. That said, at some point having a ruby dependency for a 500K node app seems silly… but at least we were consistent and clear how all of our projects were to be deployed. Any developer in the company, even if they never touched a line of node before, knew how the app was to be deployed to production.

Then came Ansible.

One of the things that always irked me about Capistrano was that it required duplication of data. Why do I need to keep a list of servers and roles in a deploy.rb file within each application when the authoritative source for that data is our provisioning tool (previously Chef-Server, now the ansible project’s inventory)? Doubly so, every time we added or removed a node from chef, I need to be sure to update the deploy.rb. There are some tools out there which attempt to link Chef and Capistrano, but none of the ones I tried worked. More worrisome was the fact that some of the steps for deployment were duplicated in chef, or Chef was shelling out to Capistrano (which required a full ruby environment) to deploy.

I’m happy to say that TaskRabbit now deploys all of our applications via Ansible, and no longer uses Capistrano. We were able to keep a homogenous command set and duplicate most of Capistrano’s features in very small amount of code. Here’s how we did it:

Server Background

• We deploy on Ubuntu 14 TLS servers.
• We have a specific user, denoted by `` in these roles.
• Our application directory structure exactly mirrors that of Capistrano (it’s a great layout), IE:

1/home/{{ deploy_user }}/www/{{ application }}/
2  - current (symlink to release)
3  - releases
4    - timestamp_1
5      - app
6      - config (symlinks to ../../shared/config)
7      - tmp (symlink to ../../shared/tmp)
8        - pids (symlink to ../../shared/pids)
9    - timestamp_2
10    - timestamp_3
11  - shared
12    - tmp
13    - config (ymls and other config files previously config'd by ansible)
14    - public
15    - cached-copy (git repo in fullt)
16    - logs
17    - pids
18    - sockets

We define inventories by RAILS_ENV (or NODE_ENV as the case may be), and then divide up each application to the sub-roles that it requires. I’ll be using the following example inventories/production file as reference:

1myApp-web1.domain.com
2myApp-web2.domain.com
3myApp-worker1.domain.com
4myApp-worker2.domain.com
5myApp-redis.domain.com
6myApp-mysql.domain.com
7
8[production]
9myApp-web1.domain.com
10myApp-web2.domain.com
11myApp-worker1.domain.com
12myApp-worker2.domain.com
13myApp-redis.domain.com
14myApp-mysql.domain.com
15
16[production:vars]
17rails_env=production
18node_env=production
19cluster_env=production
20
21[myApp]
22myApp-web1.domain.com
23myApp-web2.domain.com
24myApp-worker1.domain.com
25myApp-worker2.domain.com
26
27[myApp:unicorn]
28myApp-web1.domain.com
29myApp-web2.domain.com
30
31[myApp:resque]
32myApp-worker1.domain.com
33myApp-worker2.domain.com
34
35# ...

Playbook and API

The entry point to our deployment playbook is the deploy.yml playbook:

1- hosts: "{{ host | default(application) }}"
2  max_fail_percentage: 1
3
4  roles:
5    - { role: deploy, tags: ["deploy"], sudo: no }
6    - { role: monit, tags: ["monit"], sudo: yes }

and a rollback.yml playbook:

1- hosts: "{{ host | default(application) }}"
2  max_fail_percentage: 1
3
4  tasks:
5    - include: roles/deploy/tasks/rollback_symlink.yml
6    - include: roles/deploy/tasks/restart_unicorn.yml
7    - include: roles/deploy/tasks/restart_resque.yml

This allows us to have the following API options:

• deploy one app to all staging servers (normal use):

1ansible-playbook -i inventories/staging deploy.yml — extra-vars="application=myApp migrate=true"

• deploy one app to 1 staging server ( — limit):

1ansible-playbook -i inventories/staging deploy.yml — extra-vars="application=myApp migrate=true branch=mybranch" — limit staging-server-1.company.com

• deploy myApp production:

1ansible-playbook -i inventories/production deploy.yml — extra-vars="application=myApp migrate=true"

The beauty of the line  — hosts: "" in the playbook is that you can reference the servers in question by the group they belong to, which in our case matches the application names, and then sub-slice the group even further via optional — limit flags.

Variables

To make this playbook work, we need a collection of application metadata. This essentially mirrors the information you would provide within an application’s deploy.rb in Capistrano. However, moving this data to Ansible allows it be used not only in both of the deployment/rollback playbooks, but also in provisioning if needed. Here’s some example data for our myApp application, which we can pretend is a Rails 4 application:

From group_vars/all

1applications:
2  - myApp
3  - myOtherApp
4
5application_git_url_base: git@github.com
6application_git_url_team: myCompany
7
8deploy_email_to: everyone@myCompany.com
9
10application_configs:
11  myApp:
12    name: myApp
13    language: ruby
14    roles:
15      - unicorn
16      - resque
17    ymls:
18      - database.yml
19      - memcache.yml
20      - redis.yml
21      - facebook.yml
22      - s3.yml
23      - twilio.yml
24    pre_deploy_tasks:
25      - { cmd: "bundle exec rake assets:precompile" }
26      - { cmd: "bundle exec rake db:migrate", run_once: true, control: migrate }
27      - { cmd: "bundle exec rake db:seed", run_once: true, control: migrate }
28      - { cmd: "bundle exec rake myApp:customTask" }
29    post_deploy_tasks:
30      - { cmd: "bundle exec rake cache:clear", run_once: true }
31      - { cmd: "bundle exec rake bugsnag:deploy", run_once: true }
32
33resque_workers:
34  - name: myApp
35    workers:
36      - { name: myApp-scheduler, cmd: "resque:scheduler" }
37      - { name: myApp-1, cmd: "resque:queues resque:work" }
38      - { name: myApp-2, cmd: "resque:queues resque:work" }
39#...

You can see here that we have defined a few things:

• the configuration files needed for each app (that we place in /home/{{ deploy_user }}/www/{{ application }}/shared/config as noted above)
• metadata about the application, including the language (ruby) and the roles (unicorn and resque)
• tasks to complete before and after the "deploy". The moment the "deploy" happens here is when the symlink for the current symlink switches over.

The Role: Deploy

roles/deploy/main.yml Looks like this:

1- include: init.yml
2- include: git.yml
3- include: links.yml
4- include: config.yml
5- include: bundle.yml
6- include: pre_tasks.yml
7- include: reboot.yml
8- include: post_tasks.yml
9- include: cleanup.yml
10- include: email.yml
11- include: hipchat.yml

Lets go through each step 1-by-1:

init.yml

1- name: Generate release timestamp
2  command: date +%Y%m%d%H%M%S
3  register: timestamp
4  run_once: true
5
6- set_fact: "release_path='/home/{{ deploy_user }}/www/{{ application }}/releases/{{ timestamp.stdout }}'"
7- set_fact: "shared_path='/home/{{ deploy_user }}/www/{{ application }}/shared'"
8- set_fact: "current_path='/home/{{ deploy_user }}/www/{{ application }}/current'"
9
10- set_fact: migrate={{ migrate|bool }}
11  when: migrate is defined
12- set_fact: migrate=false
13  when: migrate is not defined
14
15- set_fact: branch=master
16  when: branch is not defined and cluster_env != 'production'
17- set_fact: branch=production
18  when: cluster_env == 'production'
19
20- set_fact: keep_releases={{ keep_releases|int }}
21  when: keep_releases is defined
22- set_fact: keep_releases={{ 6|int }}
23  when: keep_releases is not defined
24
25- name: "capture previous git sha"
26  run_once: true
27  register: deploy_previous_git_sha
28  shell: >
29    cd {{ current_path }} &&
30    git rev-parse HEAD
31  ignore_errors: true

You can see that we do a few things: — generate the release timestamp on server to use on all of them — save the paths release_path, shared_path and current_path, just like Capistrano — handle default values for the migrate, branch, and keep_releases options — learn the git SHA of the previous release

git.yml

1- name: update source git repo
2  shell: "git fetch && git reset --hard origin/master"
3  sudo: yes
4  sudo_user: "{{ deploy_user }}"
5  args:
6    chdir: "{{ shared_path }}/cached-copy"
7  when: "'{{application}}' in group_names"
8
9- name: Create release directory
10  file: "state=directory owner='{{ deploy_user }}' path='{{ release_path }}'"
11  sudo: yes
12  sudo_user: "{{ deploy_user }}"
13  when: "'{{application}}' in group_names"
14
15- name: copy the cached git copy
16  shell: "cp -r {{ shared_path }}/cached-copy/. {{ release_path }}"
17  sudo: yes
18  sudo_user: "{{ deploy_user }}"
19  when: "'{{application}}' in group_names"
20
21- name: git checkout
22  shell: "git checkout {{ branch }}"
23  sudo: yes
24  sudo_user: "{{ deploy_user }}"
25  args:
26    chdir: "{{ release_path }}"
27  when: "'{{application}}' in group_names"

This section ensure that we git-pull the latest code into the cached-copy, copy it into the new release_directory, and then checkout the proper branch

links.yml

1- name: ensure directories
2  file: "path={{ release_path }}/{{ item }} state=directory"
3  sudo: yes
4  sudo_user: "{{ deploy_user }}"
5  when: "'{{application}}' in group_names"
6  with_items:
7    - tmp
8    - public
9
10- name: symlinks
11  shell: "rm -rf {{ item.dest }} && ln -s {{ item.src }} {{ item.dest }}"
12  sudo: yes
13  sudo_user: "{{ deploy_user }}"
14  when: "'{{application}}' in group_names"
15  with_items:
16    - { src: "{{ shared_path }}/log", dest: "{{ release_path }}/log" }
17    - { src: "{{ shared_path }}/pids", dest: "{{ release_path }}/tmp/pids" }
18    - { src: "{{ shared_path }}/pids", dest: "{{ release_path }}/pids" } #Note: Double symlink for node apps
19    - {
20        src: "{{ shared_path }}/sockets",
21        dest: "{{ release_path }}/tmp/sockets",
22      }
23    - {
24        src: "{{ shared_path }}/assets",
25        dest: "{{ release_path }}/public/assets",
26      }
27    - {
28        src: "{{ shared_path }}/system",
29        dest: "{{ release_path }}/public/system",
30      }

This creates symlinks from each deployed release back to shared. This enables us to save logs, pids, etc between deploys.

config.yml

1- name: list shared config files
2  shell: "ls -1 {{ shared_path }}/config"
3  register: remote_configs
4  when: "'{{application}}' in group_names"
5
6- name: symlink configs
7  shell: "rm -f {{ release_path }}/config/{{ item }} && ln -s {{ shared_path }}/config/{{ item }} {{ release_path }}/config/{{ item }} "
8  with_items: remote_configs.stdout_lines
9  sudo: yes
10  sudo_user: "{{ deploy_user }}"
11  when: "'{{application}}' in group_names"

Here we source every file in app/shared/config/* and symlink it into app/release/config/*

bundle.yml

1- stat: path={{ release_path }}/Gemfile
2  register: deploy_gemfile_exists
3
4- name: bundle install
5  sudo: yes
6  sudo_user: "{{ deploy_user }}"
7  args:
8    chdir: "{{ release_path }}"
9  shell: >
10    bundle install
11    --gemfile {{ release_path }}/Gemfile
12    --path {{ shared_path }}/bundle
13    --without development test
14    --deployment --quiet
15  when: "'{{application}}' in group_names and deploy_gemfile_exists.stat.exists"

If there is a Gemfile in this project, we bundle install

pre_tasks.yml

1- name: deployment pre tasks (all hosts)
2  sudo: yes
3  sudo_user: "{{ deploy_user }}"
4  shell: >
5    cd {{ release_path }} &&
6    RAILS_ENV={{ rails_env }}
7    RACK_ENV={{ rails_env }}
8    NODE_ENV={{ rails_env }}
9    {{ item.cmd }}
10  run_once: false
11  when: >
12    ('{{application}}' in group_names) and
13    ({{ item.run_once | default(false) }} == false) and
14    ({{ item.control | default(true) }} != false)
15  with_items: "application_configs[application].pre_deploy_tasks"
16
17- name: deployment pre tasks (single hosts)
18  sudo: yes
19  sudo_user: "{{ deploy_user }}"
20  shell: >
21    cd {{ release_path }} &&
22    RAILS_ENV={{ rails_env }}
23    RACK_ENV={{ rails_env }}
24    NODE_ENV={{ rails_env }}
25    {{ item.cmd }}
26  run_once: true
27  when: >
28    ('{{application}}' in group_names) and
29    ({{ item.run_once | default(false) }} == true) and
30    ({{ item.control | default(true) }} != false)
31  with_items: "application_configs[application].pre_deploy_tasks"

In the application_configs part of our variable file, we defined a collection of tasks to run as part of the deploy. Here is where asset compilation would be run, etc. Note how when you define the task, we have the attributes "run_once" and "control", IE: { cmd: "bundle exec rake db:migrate", run_once: true, control: migrate }. This means that the migration task should only be run on one host, and that it should only be run when the playbook is run with the flags — extra-vars=’migrate=true’. This is how simple it is to build complex Capistrano-like roles.

reboot.yml

1- name: Update current Symlink
2  sudo: yes
3  sudo_user: "{{ deploy_user }}"
4  file: "state=link path={{ current_path }} src={{ release_path }}"
5  notify:
6    - deploy restart unicorn
7    - deploy restart resque
8  when: "'{{application}}' in group_names"
9
10- meta: flush_handlers

Now that all of our pre-tasks have been run, it’s time to actually change the deploy symlink and "restart" our applications. This simple role just changes the symlink, but the notifications are fairly involved. Some of your servers (Unicorn) may be able to gracefully restart with a simple signal, while others (like resque workers) need to fully stop and start to accept new code. Ansible makes it easy to build notification handlers that fit your needs:

handlers/main.yml

1## UNICORN ##
2
3- name: "deploy restart unicorn"
4  when: "'unicorn' in application_configs[application].roles and '{{application}}:unicorn' in group_names"
5  ignore_errors: yes
6  shell: "kill -s USR2 `cat {{ current_path }}/tmp/pids/unicorn.pid`"
7  sudo: yes
8  sudo_user: "{{ deploy_user }}"
9  notify:
10    - ensure monit monitoring unicorn
11
12- name: ensure monit monitoring unicorn
13  monit:
14    name: unicorn-{{ application }}
15    state: monitored
16  sudo: yes
17
18## RESQUE ##
19
20- name: deploy restart resque
21  ignore_errors: yes
22  shell: "kill -s QUIT `cat {{ current_path }}/tmp/pids/resque-resque-{{ item.0.name }}-{{ item.1.name }}.pid`"
23  with_subelements:
24    - resque_workers
25    - workers
26  when: "'{{ item.0.name }}:resque' in group_names and item.0.name == application"
27  notify: ensure monit monitoring resque
28  sudo: yes
29
30- name: ensure monit monitoring resque
31  monit:
32    name: "resque-{{ item.0.name }}-{{ item.1.name}}"
33    state: monitored
34  with_subelements:
35    - resque_workers
36    - workers
37  when: "'{{ item.0.name }}:resque' in group_names and item.0.name == application"
38  notify: reload monit
39  sudo: yes

You can see here that we chain notification handlers here to both restart the application and then ensure that our process monitor, monit, is configured to watch that application.

post_tasks.yml

1- name: deployment post tasks (all hosts)
2  sudo: yes
3  sudo_user: "{{ deploy_user }}"
4  shell: >
5    cd {{ release_path }} &&
6    RAILS_ENV={{ rails_env }}
7    RACK_ENV={{ rails_env }}
8    NODE_ENV={{ rails_env }}
9    {{ item.cmd }}
10  run_once: false
11  when: >
12    ('{{application}}' in group_names) and
13    ({{ item.run_once | default(false) }} == false) and
14    ({{ item.control | default(true) }} != false)
15  with_items: "application_configs[application].post_deploy_tasks"
16
17- name: deployment post tasks (single hosts)
18  sudo: yes
19  sudo_user: "{{ deploy_user }}"
20  shell: >
21    cd {{ release_path }} &&
22    RAILS_ENV={{ rails_env }}
23    RACK_ENV={{ rails_env }}
24    NODE_ENV={{ rails_env }}
25    {{ item.cmd }}
26  run_once: true
27  when: >
28    ('{{application}}' in group_names) and
29    ({{ item.run_once | default(false) }} == true) and
30    ({{ item.control | default(true) }} != false)
31  with_items: "application_configs[application].post_deploy_tasks"

post_tasks are just like pre_tasks, and allow you to run code after the servers have been restarted. Here is where you might clear caches, update CDNs, etc.

email.yml

Now the fun kicks in. Ansible makes it easy to keep adding more to your playbooks. We wanted to send the development team an email (and also notify hipchat in a similar role) every time a deploy goes out. Here’s a sample:

Here’s how to grab the variables you need:

1- name: "capture: sha"
2  run_once: true
3  register: deploy_email_git_sha
4  shell: >
5    cd {{ release_path }} &&
6    git rev-parse HEAD
7
8- name: "capture: deployer_email"
9  run_once: true
10  register: deploy_email_deployer_email
11  shell: >
12    cd {{ release_path }} &&
13    git log -1 --pretty="%ce"
14
15- name: "capture: branch"
16  run_once: true
17  register: deploy_email_branch
18  shell: >
19    cd {{ release_path }} &&
20    git rev-parse --abbrev-ref HEAD
21
22- name: "capture: commit message"
23  run_once: true
24  register: deploy_email_commit_message
25  shell: >
26    cd {{ release_path }} &&
27    git log -1 --pretty="%s"
28
29- set_fact: previous_revision='n/a'
30  when: previous_revision is defined
31
32- name: "capture: previous commits"
33  run_once: true
34  register: deploy_email_previous_commits
35  when: deploy_previous_git_sha is defined and ( deploy_previous_git_sha.stdout_lines | length > 0 )
36  shell: >
37    cd {{ release_path }} &&
38    git log {{ deploy_previous_git_sha.stdout_lines[0] }}..{{ deploy_email_git_sha.stdout_lines[0] }} --pretty=format:%h:%s --graph
39
40- name: "capture: human date"
41  run_once: true
42  register: deploy_email_human_date
43  shell: date
44
45- name: build the deploy email body
46  run_once: true
47  local_action: template
48  args:
49    src: deploy_email.html.j2
50    dest: /tmp/deploy_email.html
51
52- name: send the deploy email
53  run_once: true
54  when: no_email is not defined or no_email == false
55  local_action: shell sendmail {{ deploy_email_to }} < /tmp/deploy_email.html

and our email template is:

1From: {{ deploy_email_deployer_email.stdout_lines[0] }}
2Subject: Deployment: {{ application }} [ {{ cluster_env }} ]
3Content-Type: text/html
4MIME-Version: 1.0
5
6<h1>
7  <a href="https://github.com/{{ application_git_url_team }}/{{ application }}">{{ application }}</a>
8  was deployed to {{ cluster_env }} by {{ deploy_email_deployer_email.stdout_lines[0] }}
9  at {{ deploy_email_human_date.stdout_lines[0] }}
10</h1>
11
12<h2>The {{ deploy_email_branch.stdout_lines[0] }} branch was deployed to {{ vars.play_hosts | count }} hosts</h2>
13<p>The latest commit is: <a href="https://github.com/{{ application_git_url_team }}/{{ application }}/commit/{{ deploy_email_git_sha.stdout_lines[0] }}">{{ deploy_email_commit_message.stdout_lines[0] }}</a> </p>
14
15<strong>Hosts:</strong>
16<ul>
17{% for host in vars.play_hosts %}
18  <li>{{ host }}</li>
19{% endfor %}
20</ul>
21
22{% if deploy_email_previous_commits is defined and deploy_previous_git_sha.stdout_lines | length > 0 %}
23<strong>New on these servers since the last deploy:</strong>
24<br />
25{% for line in deploy_email_previous_commits.stdout_lines %}
26  {{ line }}<br />
27{% endfor %}
28
29{% endif %}

And that’s how you build Capistrano within Ansible! You can see how simple it is to translate a complex tool into a few hundred lines of Ansible… with very clear responsibilities and separation. It’s also very easy to extend this to fit your workflow.