Ansible - create a pub/priv AWS VPC with NAT instance

Jun 15, 2015   #ansible  #aws  #vpc 

Today I had a requirement to spin up an AWS VPC with Public and Private subnets. This includes a NAT instance, and custom routing tables.

To start with, I create a VPC without a routing table. Once the VPC is up, I can add a NAT instance, and then update the VPC.

---
# play.yml
- hosts: localhost
  vars:
    vpc_list:
      - region: eu-west-1
        state: present
        cidr_block: 10.1.0.0/16
        resource_tags:
          Environment: dev 
        internet_gateway: True
        # there are 3 zones in eu-west-1 (Ireland)
        subnets:
          - cidr: 10.1.0.0/24
            az: eu-west-1a
            resource_tags: { "Name": "dev_public", "Environment": "dev", "Tier": "public" }
          - cidr: 10.1.100.0/24
            az: eu-west-1a
            resource_tags: { "Name": "dev_private", "Environment": "dev", "Tier": "private" }
    nat_list:
      - region: eu-west-1
        keypair: ec2 
        instance_type: "t2.small" 
        image: "ami-ef76e898" # amzn-ami-vpc-nat-hvm-2015.03.0.x86_64-ebs
        instance_tags: { "Name": "dev_nat", "Environment": "dev" }
        exact_count: 1
        count_tag: { "Name": "dev_nat" }

  tasks:

    - name: process vpc
      ec2_vpc:
        region: "{{ item.region }}" 
        state: "{{ item.state }}" 
        cidr_block: "{{ item.cidr_block }}" 
        resource_tags: "{{ item.resource_tags }}" 
        internet_gateway: "{{ item.internet_gateway }}" 
        subnets: "{{ item.subnets }}" 
      with_items: vpc_list
      register: ec2_vpc_out

The ec2 module, takes a vpc_subnet_id param so it can place the instance, which means I need to retrieve the one I want from the ec2_vpc results - which looks a bit like this

        "ec2_vpc_out": {
            "changed": true, 
            "msg": "All items completed", 
            "results": [
                {
                    #...
                    "subnets": [
                        {
                            "az": "eu-west-1a", 
                            "cidr": "10.1.0.0/24", 
                            "id": "subnet-f6275193", 
                            "resource_tags": {
                                "Environment": "dev", 
                                "Name": "dev_public", 
                                "Tier": "public"
                            }
                        }, 
                        {
                            "az": "eu-west-1a", 
                            "cidr": "10.1.100.0/24", 
                            "id": "subnet-f1275194", 
                            "resource_tags": {
                                "Environment": "dev", 
                                "Name": "dev_private", 
                                "Tier": "private"
                            }
                        }
                    ], 
                    #...

Looking at that, I couldn’t see an easy way to get the id that matched the subnet I wanted, so I used ansible’s awesome plugability and made one.

A custom filter that allowed me to pass the list of subnets, and given a tag key and value, return those that match.

# ansible_plugins/filter_plugins/get_subnets.py
from jinja2.utils import soft_unicode

def get_subnets(value, tag_key, tag_value, return_key='id'):
    # return an attribute for all subnets that match
    subnets = []
    for item in value:
      for key, value in item['resource_tags'].iteritems():
        if key == tag_key and value == tag_value:
          subnets.append(item[return_key])
            
    return subnets


class FilterModule(object):
    ''' Ansible core jinja2 filters '''

    def filters(self):
        return {
            'get_subnets': get_subnets,
        }

This allowed me to create the NAT instance as follows

    - name: process all ec2 instances
      ec2:
        region: "{{ item.0.region }}" 
        keypair: "{{ item.0.keypair }}" 
        instance_type: "{{ item.0.instance_type }}" 
        image: "{{ item.0.image }}" 
        instance_tags: "{{ item.0.instance_tags }}" 
        exact_count: "{{ item.0.exact_count }}" 
        count_tag: "{{ item.0.count_tag }}" 
        vpc_subnet_id: "{{ item.1.subnets | get_subnets('Tier', 'public') | first }}"
        wait: yes 
        wait_timeout: 500 
      with_together:
        - "{{ nat_list }}" 
        - "{{ ec2_vpc_out.results }}" 
      register: ec2_out

Next I needed to update the VPC, adding the internet gateway as the default route for the public tier, and the nat instance as the default route for the private tier.

      - name: update vpc routing tables
        ec2_vpc:
          region: "{{ item.0.region }}"
          state: "{{ item.0.state }}"
          cidr_block: "{{ item.0.cidr_block }}"
          resource_tags: "{{ item.0.resource_tags }}"
          internet_gateway: "{{ item.0.internet_gateway }}"
          subnets: "{{ item.0.subnets }}"
          route_tables:
            - subnets: "{{ item.0.subnets | get_subnets('Tier', 'public', 'cidr') }}"
              routes:
                - dest: 0.0.0.0/0
                  gw: igw
            - subnets: "{{ item.0.subnets | get_subnets('Tier', 'private', 'cidr') }}"
              routes:
                - dest: 0.0.0.0/0
                  gw: "{{ item.1.instance_ids[0] }}"
        with_together:
          - "{{ vpc_list }}"
          - "{{ ec2_out.results }}"
        register: ec2_vpc_out

Note: the ec2_vpc module takes actual cidr’s for the route table subnets,_ not _subnet id’s (as used by the ec2 module), which is why my filter above allows a different key to be specified for the return value.

Unfortunately I would have preferred to not repeat all the ec2_vpc params, however only some of them are allowed to be ignored (and for some reason leaving out the subnets didn’t work), so I just re-wrote the entire thing and added the routing table section.

The above works for the most part, in the next few days, I’ll be cleaning it up and extending it with a security group (required for the private instances to use the NAT instance).

_Note: the example I’ve used has a vpc_list and natlist, the idea being I can easily extend to multiple environments and/or regions

Note: I use the ~/.boto config file or ENV variables for setting the AWS Access Key and AWS Secret Key