How to Manage and Automate AWS with Ansible

Ansible is an open-source and battle-tested automation tool with simplicity and powerful capabilities. These qualities make it an excellent choice for configuration management, infrastructure provisioning, and application deployment use cases. 

Leveraging Ansible to manage and provision cloud infrastructure on AWS enables operations to streamline and automate various tasks. It enables cross-platform automation and orchestration at scale and is considered an excellent option for configuration management, infrastructure provisioning, and application deployment use cases. 

What is Ansible used for in AWS?

Ansible is used in AWS for various purposes, including configuration management, CI/CD and application deployment, and cloud provisioning and management. It also supports network automation, security and compliance automation, disaster recovery automation, and complex workflow automation, making it a versatile tool for automating and streamlining various AWS operations.

Benefits of using Ansible in AWS

Ansible's robust automation and orchestration capabilities at scale make it a valuable tool in every Cloud Engineer's toolbelt. Below are the most important benefits of using Ansible in AWS:

1. The idempotent nature of Ansible ensures that tasks are executed consistently and predictably, reducing the risk of operations. 
2. Ansible does not require additional software to be installed on the managed nodes, making it lightweight and easy to get started. 
3. Ansible's modular architecture enables extensibility and customization to cover even the most demanding needs by developing custom Ansible modules. 

Ansible AWS Tutorial: How to use Ansible in AWS?

In this section, we'll show you how to configure and use Ansible with AWS. We'll guide you through the necessary steps to set up Ansible, connect it to your AWS environment, and automate various AWS tasks.

1. Connect to AWS services

To use Ansible and target resources on AWS, you must first set up authentication. The most common methods include using environment variables, AWS CLI configuration and profiles, Ansible Vault, and costing credentials in the Ansible Playbooks.

To set environment variables for authentication purposes, export the AWS secret access key and AWS access key ID:

export AWS_ACCESS_KEY_ID='<YOUR ACCESS_KEY_ID>'
export AWS_SECRET_ACCESS_KEY='>YOUR_SECRET_ACCESS_KEY>'

Another option is to use the official AWS CLI to configure a profile and persist these values in a local AWS credentials file. After installing the AWS CLI, check out the Set up the AWS CLI to configure this based on your preferred method. 

Let's say we want to configure some long-term credentials locally in this case. Type aws configure and specify values for access key ID, secrets access key, and AWS region. Alternatively, we can request short-term credentials, assume an AWS IAM role, or use the IAM Identity Center (SSO). 

You could also store credentials inside the Ansible Playbooks with variables. Make sure to avoid hardcoding AWS secrets and keys inside playbooks. If you wish to go down that way, leverage Ansible Vault to encrypt these values before using them. 

For example, let's say we store our secrets in secrets.yml:

secrets.yml

aws_access_key_id: my-aws-access-key-id
aws_secret_key: my-aws-secret-access-key

Then, you can encrypt this file with the following:

ansible-vault encrypt secrets.yaml

To use this encrypted file within a playbook:

playbook.yml

- hosts: localhost
  vars_files:
    - secrets.yml
  tasks:
    - ec2_instance_info:
        aws_access_key_id: "{{ aws_access_key_id }}"
        aws_secret_key: "{{ aws_secret_key }}"`

and trigger the Ansible playbook while passing the password when prompted:

ansible-playbook playbook.yml --ask-vault-pass

2. Provision the AWS infrastructure

While we know that it is not a best practice to provision infrastructure using Ansible, it is still possible.

Typically, provisioning AWS infrastructure with Ansible involves creating and managing different resources via Ansible playbooks. 

In this example, we are creating an EC2 instance to deploy a web application. We would also need a VPC, a subnet, security groups, and an Internet Gateway attached to the VPC.

ec2_playbook.yml

- name: Create EC2 and necessary AWS resources
  hosts: localhost
  gather_facts: no
  vars:
    region: us-east-1
    instance_type: t2.micro
    ami_id: ami-04e5276ebb8451442
    vpc_cidr_block: 10.0.0.0/16
    subnet_cidr_block: 10.0.1.0/24
    security_group_cidr_ingress: 0.0.0.0/0
    security_group_cidr_egress: 0.0.0.0/0
    security_group_ports:
        - 80
        - 443

  tasks:
    - name: Create VPC
      ec2_vpc_net:
        name: MyVPC
        cidr_block: "{{ vpc_cidr_block }}"
        region: "{{ region }}"
        tags:
          Name: MyVPC
      register: my_vpc

    - name: Output VPC ID
      debug:
        msg: "VPC ID is {{ my_vpc.vpc.id }}"

    - name: Create subnet
      ec2_vpc_subnet:
        state: present
        vpc_id: "{{ my_vpc.vpc.id }}"
        cidr: "{{ subnet_cidr_block  }}"
        region: "{{ region }}"
        tags:
          Name: MySubnet
      register: my_subnet

    - name: Output Subnet ID
      debug:
        msg: "Subnet ID is {{ my_subnet.subnet.id }}"

    - name: Create internet gateway
      ec2_vpc_igw:
        vpc_id: "{{ my_vpc.vpc.id }}"
        region: "{{ region }}"
        tags:
          Name: MyIGW
      register: igw

    - name: Output Internet Gateway ID
      debug:
        msg: "Internet Gateway ID is {{ igw.gateway_id }}"

    - name: Create security group
      ec2_group:
        name: "MySc"
        description: "My security group"
        vpc_id: "{{ my_vpc.vpc.id }}"
        region: "{{ region }}"
        rules:
          - proto: tcp
            ports: " {{ security_group_ports }"
            cidr_ip: "{{ security_group_cidr_ingress }}"
        rules_egress:
          - proto: all
            cidr_ip: "{{ security_group_cidr_egress }}"
      register: security_group

    - name: Output Security Group ID
      debug:
        msg: "Security Group ID is {{ security_group.group_id }}"

    - name: Launch instance
      ec2_instance:
        name: "MyInstance"
        instance_type: "{{ instance_type }}"
        region: "{{ region }}"
        image_id: "{{ ami_id }}"
        subnet_id: "{{ my_subnet.subnet.id }}"
        wait: yes
        security_group: "{{ security_group.group_id }}"
        network:
          assign_public_ip: true
        tags:
          Environment: Testing
      register: ec2

    - name: Output Instance Details
      debug:
        msg: "Instance ID is {{ ec2.instances[0].instance_id }}"

To go ahead and create these AWS resources, run the playbook.

ansible-playbook ec2_playbook.yml

PLAY [Create EC2 and necessary AWS resources] **********************************************************************************************************************************************************************************************************************************

TASK [Create VPC] **************************************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Output VPC ID] ***********************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "VPC ID is vpc-008c5e6b9d3f31028"
}

TASK [Create subnet] ***********************************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Output Subnet ID] ********************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Subnet ID is subnet-00557cf19209f323e"
}

TASK [Create internet gateway] *************************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Output Internet Gateway ID] **********************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Internet Gateway ID is igw-0371eefa567f9fdbe"
}

TASK [Create security group] ***************************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Output Security Group ID] ************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Security Group ID is sg-0806f59420e25294f"
}

TASK [Launch instance] *********************************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Output Instance Details] *************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Instance ID is i-09f60c41a4a3bc583"
}

PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=10   changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

3. Configure dynamic host inventory in AWS

An Ansible inventory is a collection of managed hosts we want to manage with Ansible for various automation and configuration management tasks. Typically, when starting with Ansible, we define a static list of hosts known as the inventory. 

As many modern environments are dynamic, cloud-based, and possibly spread across multiple providers, maintaining a static list of managed nodes is time-consuming, manual, and error-prone. Ansible's dynamic inventory feature allows you to automatically pull inventory from external sources, such as cloud providers like AWS.

To track and target a dynamic set of hosts, you can use the aws_ec2 inventory plugin. This way, you can query AWS for instances and organize them into groups that can be targeted in your playbooks. The dynamic inventory plugin connects to AWS, fetches details about the instances, and outputs these details in a JSON format that Ansible can understand.

dynamic_inventory_aws_ec2.yml

plugin: aws_ec2
regions:
  - us-east-1
hostnames: tag:Name
keyed_groups:
  - key: tags['Environment']
    prefix: env
  - key: tags['Role']
    prefix: role

and in your ansible.cfg file add:

ansible.cfg

[inventory]
enable_plugins = aws_ec2

Then you can view the hosts of the dynamic inventory grouped as we specified with:

ansible-inventory -i dynamic_inventory_aws_ec2.yml --graph
@all:
  |--@ungrouped:
  |--@aws_ec2:
  |  |--worker1
  |  |--MyInstance
  |  |--db1
  |  |--Web1
  |--@env_Testing:
  |  |--worker1
  |--@role_Worker:
  |  |--worker1
  |--@env_Production:
  |  |--MyInstance
  |  |--Web1
  |--@role_Web:
  |  |--MyInstance
  |  |--Web1
  |--@env_Staging:
  |  |--db1
  |--@role_Database:
  |  |--db1

If you want to filter for specific tags, use the filters argument. This example will only show you the instances tagged with Environment: Testing:

# You can include only instances with specific tags, or all instances
filters:
  tag:Environment: Testing  # Only include instances with these tags`

When running your playbook, you can specify the inventory file:

ansible-playbook -i dynamic_inventory_aws_ec2.yml playbook.yml

To target only a specific group of instances, for example, only the Production instances, use the --limit argument:

ansible-playbook -i dynamic_inventory_aws_ec2.yml playbook.yml --limit 'env_Production'

💡 You might also like:

• How to Manage Kubernetes with Ansible
• Configuration Management with Ansible
• Using Terraform & Ansible together

4. Leverage Ansible for configuration management at scale

Another way we can leverage Ansible in AWS is for configuration management needs of our fleet of instances or other cloud services. When it comes to managing large-scale cloud infrastructures, Ansible's simplicity is a significant benefit. 

Ansible can run commands on multiple hosts simultaneously or in batches. There is an option for long-running tasks to allow asynchronous execution, allowing other tasks to proceed without waiting for the previous one.

When you need to deploy a web server across many EC2 instances, you could create an Ansible playbook that helps you configure them. 

Here's an example of a playbook you could use to install Nginx to Ubuntu instances.

configure_web_server.yml

- name: Create EC2 and necessary AWS resources
  hosts: all
  become: true
  gather_facts: no
  vars:
    nginx_version: 1.24.0-2ubuntu7
    nginx_custom_directory: /home/ubuntu/nginx

  tasks:
    - name: Update and upgrade apt
      ansible.builtin.apt:
        update_cache: yes
        cache_valid_time: 3600
        upgrade: yes

    - name: "Install Nginx to version {{ nginx_version }}"
      ansible.builtin.apt:
        name: "nginx={{ nginx_version }}"
        state: present

    - name: Copy the Nginx configuration file to the host
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/sites-available/default

    - name: Create link to the new config to enable it
      file:
        dest: /etc/nginx/sites-enabled/default
        src: /etc/nginx/sites-available/default
        state: link

    - name: Create Nginx directory
      ansible.builtin.file:
        path: "{{ nginx_custom_directory }}"
        state: directory

    - name: Copy index.html to the Nginx directory
      copy:
        src: index.html
        dest: "{{ nginx_custom_directory }}/index.html"
      notify: Restart the Nginx service

  handlers:
    - name: Restart the Nginx service
      service:
        name: nginx
        state: restarted

Once the playbook is configured, run the below Ansible command to target all the EC2 instances that are tagged with "Environment: Staging" using the dynamic inventory concept that we introduced above:

ansible-playbook -i dynamic_inventory_aws_ec2.yml configure_web_server.yml -- limit env_Staging

5. Use Ansible for AWS image building

After configuring an EC2 instance with Ansible, you can also register an EC2 AMI image with the configuration persisted. To create an AMI from a running or stopped EC2 image, leverage the amazon.aws.ec2_ami module. 

For example, at the end of our playbook ec2_playbook.yml introduced above to create an EC2 instance, add these two tasks:

- pause:
        minutes: 5

    - name: Creating the AMI from of the instance
      ec2_ami:
        instance_id: "{{ ec2.instances[0].instance_id }}"
        wait: yes
        name: "{{ ami_name }}"
        tags:
          Name: "{{ ami_name }}"

Here, we added a custom pause task to wait for the EC2 instance be be in running state before creating the AMI. To create the AMI run:

ansible-playbook ec2_playbook.yml

6. Run Ansible playbooks with the AWS systems manager

AWS Systems Manager gathers all the operational sub-services for your AWS applications and resources for secure, end-to-end management solutions for hybrid and multicloud environments at scale.

The Systems Manager provides robust support for running Ansible playbooks with the integration of an SSM document called AWS-ApplyAnsiblePlaybooks. This feature supports fetching Ansible manifests from the GitHub repository or S3 buckets and complex playbooks.

To use a playbook or a role with the Systems Manager, we have to create a State Manager association, which is basically a configuration that defines the state that you want to maintain on your resources.

For this example, we created an S3 bucket and uploaded our configure_web_server.yml there. Then, using the command below and targeting our AWS account, we can register this association to run our playbook.

Change the placeholder <your_S3_bucket_name> in the following command with your S3 bucket name. We are targeting the us-east-1 region and only the EC2 instances with the tag role:webserver.

aws ssm create-association --name "AWS-ApplyAnsiblePlaybooks" --parameters '{"SourceType":["S3"], "SourceInfo":["{\"path\": \"https://<your_S3_bucket_name>.s3.amazonaws.com/configure_web_server.yml\"}"], "InstallDependencies":["True"], "PlaybookFile":["ec2_playbook.yml"], "ExtraVariables":["myregion=us-east-1"], "Check":["False"], "Verbose":["-v"]}' --targets '[{"Key":"tag:role","Values":["webserver"]}]' --max-concurrency "50" --max-errors "0"

With this SSM association generated, we can automate Ansible playbook deployment via Systems Manager on desired instances on demand or at a predefine schedule. 

7. Clean up

To prevent unintentional AWS costs, ensure you terminate any EC2 instances, the Internet Gateway, and the SSM association you launched through the examples of this blog post. 

You can do this manually through the console or create a separate playbook to clean up and delete the AWS resources when they are no longer needed. Including provisioning and de-provisioning tasks in your Ansible playbooks is a good practice.

Best practices for using Ansible with AWS

This section presents key best practices for using Ansible with AWS for your automation and cloud management needs.

Security

• Avoid hardcoding sensitive information in your playbooks. Use environment variables or the AWS credentials file. 
• For EC2 instances running Ansible, consider using IAM roles that provide necessary permissions without using static credentials.
• Don't store sensitive values in plain text; for secrets and sensitive values, use Ansible Vault to encrypt variables and files and protect sensitive information. 

Inventory

• Leverage the Dynamic Inventory option. Instead of hardcoding hosts in your Ansible inventory, use the AWS EC2 dynamic inventory. This allows Ansible to automatically query AWS for running instances and use those instances based on tags, regions, and other attributes. It keeps your inventory updated as the state of AWS resources changes.
• Leverage Dynamic Grouping at runtime. Using the group_by module based on a specific attribute, we can create dynamic groups. For example, group hosts dynamically based on their operating system and run different tasks on each without defining such groups in the inventory.

Automation & execution

• Leverage the specific AWS Cloud Modules; Ansible includes a wide range of modules specifically designed for AWS, such as EC2, S3, RDS, and more. These Ansible AWS modules are designed to interact efficiently with AWS services, reducing the need to script these operations manually.
• Ensure your playbooks are idempotent. Running them multiple times on the same system will produce the same result without unintended side effects. This is crucial for maintaining consistent state management and reducing errors during repeated provisioning or configuration updates.
• To improve the performance of your playbooks, explore strategies like asynchronous actions and polling to manage long-running tasks and batch operations to reduce overhead when interacting with AWS APIs.
• Consider using an infrastructure management and collaboration tool such as Spacelift to manage the complexities and compliance challenges of using Ansible at scale. Check out the Getting Started guide.

If you are looking for generic Ansible best practices, check out the 44 Ansible Best Practices to Follow [Tips & Tricks] blog post.

How Spacelift can help you with Ansible projects?

Spacelift's vibrant ecosystem and excellent GitOps flow can greatly assist you in managing and orchestrating Ansible. By introducing Spacelift on top of Ansible, you can then easily create custom workflows based on pull requests and apply any necessary compliance checks for your organization.

Another great advantage of using Spacelift is that you can manage different infrastructure tools like Ansible, OpenTofu, Terraform, Pulumi, AWS CloudFormation, and even Kubernetes from the same place and combine their Stacks with building workflows across tools. Spacelift greatly simplifies and elevates your workflow for all of these tools, and the capability of creating dependencies between stacks and passing outputs gives you the possibility to make end-to-end deployments while just making a small change.

If you want to learn more about Spacelift working with Ansible, check our documentation, read our Ansible guide or book a demo with one of our engineers.

Key points

In this blog post, we delved into using Ansible in AWS environments. We explored the benefits of leveraging Ansible for configuration management and automation of cloud resources and went through detailed examples of using Ansible to create and manage resources on AWS and automate Ansible playbook deployment. Finally, we discussed a few best practices, such as using dedicated AWS related modules, to take your Ansible AWS game to the next level.

Thanks for reading, and I hope you enjoyed this article as much as I did.

Written by Ioannis Moustakis