Use Ansible dynamic inventory for AWS Infrastructure as Code automation

In my previous article https://bit.ly/2Xr7iSv I explored how to use Terraform to automate infrastructure provisioning on AWS, now I want to build on that and see how to go a step forward in the process and implement a fully automated process for configuring infrastructure and deploying applications to AWS, extending the Infrastructure as Code approach by adopting Ansible (https://www.ansible.com/).

At the end of this article you will gain an understanding of:

• how to use Ansible to configure EC2 instances that are not publicly accessible, using a Bastion Host and configuring SSH appropriately;
• how Ansible dynamic inventory allows to manage highly dynamic infrastructures as AWS and other Cloud providers.

Before you start

You can just read through this article to gain an understanding about the technical elements you need to implement an Infrastructure as Code approach and how Ansible can be effectively adopted in a dynamic environment such as AWS; in this case you can skip this paragraph and go straight to Architecture chapter and go on from there.

If you prefer to follow along and actually try the technologies in your own environment, I encourage you to first read my previous article on this topic (https://bit.ly/2Xr7iSv) because it lays down the necessary foundation; you will also need to install these other utilities on your workstation:

1. Install Ansible command line utility : you can find detailed instructions at the link here;
2. Install Angular command line utility : follow the instructions at the link here to setup Angular and all its prerequisites.

Architecture

Let me briefly recap the overall scenario, based on a simple application (my Windfire Restaurant application), made of just 2 components, one rendering the UI (developed in Angular) and the other exposing a service layer (developed in Node.js).

Figure 1 — Windfire Restaurants application component architecture

The infrastructure architecture that supports the application is the one in the figure below, defined in an AWS Single Availability Zone.

How the infrastructure is defined, configured and provisioned, by using Terraform, is described in my previous article https://bit.ly/2Xr7iSv and will not be explained again here.

Figure 2 — AWS Single Availability Zone architecture

The network infrastructure is designed with the following characteristics:

• 1 Frontend Subnet where 1 Web Server gets instantiated: Network ACL and Security group allow HTTP/HTTPS ingress from the internet and allow SSH ingress connection (TCP on port 22) from Bastion Security Group only.
• 1 Backend Subnet where 1 Backend Server gets instantiated: Network ACL and Security group allow HTTP ingress on port 8082 from the internet and allow SSH ingress connection (TCP on port 22) from Bastion Security Group only.
• 1 Management Subnet where 1 Bastion Host gets instantiated: Network ACL and Security group allow SSH ingress connection (TCP on port 22) from your own workstation IP only and allow SSH egress connection only to Frontend and Backend Security Groups.

If you want to follow along, start by cloning and have a look at the content of the following GitHub repositories, where you can find all the code to apply Infrastructure as Code provisioning and application deployment.

cd $HOME
git clone https://github.com/robipozzi/windfire-restaurants-devops.git
git clone https://github.com/robipozzi/windfire-restaurants-ui.git
git clone https://github.com/robipozzi/windfire-restaurants-node.git

Having already provisioned the infrastructure, let’s go straight to the topic.

Application deployment with Ansible

Provisioning the underlying infrastructure on a Cloud Provider is a great thing but, in the end, is just the tip of the iceberg; once you have done that you need to configure the servers, install and configure software, deploy application specific code, gluing all together in something that really works towards your application goals.

Ansible is a great piece of technology to do all that and uses SSH to connect to an inventory of hosts and run tasks on them in a repeatable and consistent way.

Wait, wait, did I say SSH? As I mentioned above my Web Server and Backend Server are running with their own Security Groups that, for good security reasons, prevent SSH connection from the Internet and this simply means I cannot run Ansible from my workstation and reach them, this is a big trouble.

Luckily there is a solution (looks like there is one almost everytime) and that is SSH from your workstation to the servers, jumping through a “Bastion Host”, a server which is accessible from your workstation (and only from that) and which have access to the destination servers (have a look at Figure 4).

Running Ansible through a Bastion Host

The article Running Ansible Through an SSH Bastion Host explains the concept and how to apply it to Ansible very well, thank you very much @scott_lowe, all the credits go to him.

There are two configuration files you will need to create to make the whole thing work:

1. An Ansible specific SSH configuration file that defines the SSH bastion host and other parameters.
2. Ansible configuration file that declare the usage of the custom SSH configuration file.

Let’s see them one by one.

Ansible SSH Configuration File

Below there is my sample SSH configuration file (I called it ansible-ssh.cfg); to create it you will need to find out values that depends on your specific environment:

• the subnet CIDR where your servers will be instantiated (in my case it is 10.0.0.0/16, or 10.0.*.*)
• Public IP for your Bastion Host, as assigned by AWS when it gets created (18.156.82.56 in my case)
• Public Hostname for your Bastion Host, as assigned by AWS when it gets created (ec2–18–156–82–56.eu-central-1.compute.amazonaws.com in my case)
• The SSH key that will be shared (windfire-aws-key.pem in my case)
• the user (ec2-user, which is the default for AWS)

Figure 7 — Ansible custom SSH configuration

What does this configuration file do? It is quite easy, once you know and understand it -:)

• The Host 10.10.*.* line indicates that all hosts in those subnets will use the settings defined in that block; specifically, all hosts will be accessed using the ProxyCommand setting and connect through 18.156.82.56 (the Bastion Host) using the specified private key (that must be created and available in ~/.ssh/windfire-aws-key.pem).
• The Host 18.156.82.56 combines settings for acting as an SSH bastion host with settings for using SSH multiplexing (the ControlMaster, ControlPath, and ControlPersist settings).

Create and store this file wherever you like, but make note of where the file is stored as you’ll need it for the next step: configuring Ansible to use the custom SSH settings.

Ansible Configuration File

Once you created the custom SSH configuration file you will need to explicitly tell Ansible to use those settings when connecting to the managed hosts. This is accomplished by creating (or modifying) an Ansible configuration file (I called it ansible-aws.cfg) with the following settings:

Figure 8 — Ansible configuration

We will go back to [inventory] section in a minute, for now just look at [ssh_connection] section, here is where you instruct Ansible to use the SSH configuration settings, that you defined before, to connect to the target hosts.

Ansible Dynamic Inventory Plugin

There is a final thing you need to consider: Ansible usually reads the hosts it needs to connect from an inventory file, but how can you do this in such a dynamic environment like AWS where you do not have a predefined static list of servers? Again, there is a solution to this problem and it relies on Ansible Inventory plugin mechanism , which allows to connect to a Cloud Provider and dynamically get the host list.

It is fairly easy, you just need 2 things, one is to enable the dynamic inventory in Ansible configuration file, just like this

[inventory]
enable_plugins = auto, yaml, ini

And, second, you need to create a file called windfire.aws_ec2.yaml (you can call it whatever you want, it needs just to end with *.aws_ec2.yaml suffix)

Figure 9 — Ansible dynamic inventory plugin configuration for AWS

This file instructs Ansible to connect to AWS, search for a Tag named “Role” on EC2 instances and construct the host inventory on the fly.

Now you are all set and you can finally launch your Ansible playbook, just be aware to:

• set ANSIBLE_CONFIG environment variable appropriately to point to the location where you saved your ansible-aws.config;
• set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in order for Ansible to authenticate to AWS and get the dynamic host inventory

To test that the whole thing works, I provided a simple playbook that just pings the provisioned servers, you can try with the following:

cd $HOME/windfire-restaurants-ui
export AWS_ACCESS_KEY_ID=<YOUR_AWS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET>
export ANSIBLE_CONFIG=<POINT_TO_YOUR_ANSIBLE_CONFIG_FILE_LOCATION>
ansible-playbook -i deployment/aws/windfire.aws_ec2.yaml deployment/aws/ping.yaml

In my GitHub repositories I also put some scripts that pull together everything and fully automate the infrastructure provisioning and application deployment process.

You can try it by running the provision.sh script (available at this GitHub Link), with -f option (for fullstack provisioning and deployment); the script will require your AWS Key ID and Secret

cd $HOME
git clone https://github.com/robipozzi/windfire-restaurants-devops.git
git clone https://github.com/robipozzi/windfire-restaurants-ui.git
git clone https://github.com/robipozzi/windfire-restaurants-node.git
cd $HOME/windfire-restaurants-devops/aws/SingleZone
./provision.sh <YOUR_AWS_KEY_ID> <YOUR_AWS_SECRET> -f

And you can verify everything works by simply open a browser and hit this URL: http://<YOUR_AWS_WEBSERVER_HOSTNAME>

Figure 10— Application Home Page