Dynamic AWS Inventory in Ansible: Keep Playbooks in Sync with Your Cloud

Problem: Static Inventories Drift Out of Sync

When you run an Ansible playbook against a cloud environment, the inventory file is usually a static list of host names or IPs. In a dynamic cloud, instances are spun up and terminated all the time. Updating a hosts file manually is error‑prone and defeats the purpose of automation. The result? Playbooks that target the wrong hosts, failures that are hard to diagnose, and configuration drift that undermines security and compliance.

Why Dynamic Inventory Matters

Ansible’s dynamic inventory feature lets a script query AWS at runtime and build a JSON inventory on the fly. The aws_ec2.py script (or the aws_ec2.yml plugin in newer Ansible releases) talks to EC2’s DescribeInstances API, pulls instance metadata, and returns a structured inventory that Ansible can consume immediately. Because the inventory is refreshed on every ansible‑inventory or playbook run, you always operate on the current set of instances.

Key benefits:

• Eliminates manual inventory maintenance.
• Supports filtering by tags, instance state, security groups, etc.
• Works with Ansible’s group_by plugin to auto‑create logical groups.
• Reduces configuration drift and manual errors.

Setting Up AWS Dynamic Inventory

Prerequisites:

• Python 3.6+ and boto3 installed on the Ansible control node.
• Valid AWS credentials with ec2:DescribeInstances permission.
• Ansible 2.9+ (the inventory plugin approach is preferred over the legacy script).

1. Install boto3 if missing:

pip install boto3

2. Create or edit ansible.cfg to point to the AWS inventory plugin:

[defaults]
inventory = aws_ec2.yml
host_key_checking = False

3. Create aws_ec2.yml with the plugin configuration. Replace placeholders with your values:

plugin: aws_ec2
regions:
  - us-east-1
  - us-west-2
  # Add more regions as needed
filters:
  instance-state-name: running
  tag:Environment: production
  # Adjust filters to narrow down the host set
keyed_groups:
  - key: tags.Name
    prefix: name
  - key: tags.App
    prefix: app

Explanation of key sections:

• regions – list of AWS regions to query.
• filters – EC2 filters; you can use any valid EC2 filter key.
• keyed_groups – automatically create inventory groups based on tag values.

4. Verify the inventory is working:

ansible-inventory --list --graph

Expected output will show a tree with your regions, groups, and host names. If the output is empty, check that boto3 is installed and that the AWS credentials have the necessary permissions.

Using Filters & Groups in Playbooks

With the inventory in place, you can target hosts by group names that map to tags or other metadata. For example, to install Nginx on all instances tagged with App=web:

- hosts: app_web
  become: true
  tasks:
    - name: Install Nginx
      apt:
        name: nginx
        state: present

Because the group app_web is automatically created by keyed_groups, the playbook will always apply to the current set of instances matching that tag, even if new instances are launched or old ones are terminated.

Practical Example: Rolling Out a Configuration Change

Assume you need to update the /etc/hosts file on all web servers in the production environment. Here’s a minimal playbook that uses the dynamic inventory:

---
- hosts: app_web
  become: true
  vars:
    hosts_entries:
      - { ip: 10.0.0.5, host: app1.example.com }
      - { ip: 10.0.0.6, host: app2.example.com }
  tasks:
    - name: Ensure /etc/hosts entries
      lineinfile:
        path: /etc/hosts
        line: "{{ item.ip }} {{ item.host }}"
        state: present
      loop: "{{ hosts_entries }}"

Run the playbook:

ansible-playbook -i aws_ec2.yml update_hosts.yml

Because the inventory is dynamic, if a new instance is launched with the App=web tag, the next run will automatically include it in the target group. No manual inventory edits are required.

Trade‑offs & Limitations

• Dependency on boto3: The plugin requires Python 3 and the boto3 library. Missing dependencies cause the inventory to fail silently.
• IAM Permissions: The credentials used by Ansible must allow ec2:DescribeInstances. Insufficient permissions result in an empty inventory.
• API Rate Limits: In very large accounts, the inventory script may hit AWS API limits. Consider adding max_results or using pagination settings in the plugin config.
• Latency: Inventory generation occurs at the start of each playbook run, adding a few seconds of overhead.
• Complex Filtering: While filters are powerful, overly complex filter sets can produce confusing results or miss hosts if tags are inconsistent.

Mitigation tips:

• Use a dedicated IAM role or user with minimal required permissions.
• Cache inventory output if you run playbooks frequently against the same set of hosts.
• Validate the inventory with ansible-inventory --list before running critical playbooks.

Next Steps

Now that you have a live, filter‑able inventory, you can:

• Integrate with CI/CD pipelines to automatically deploy to newly created instances.
• Extend the plugin configuration to include other AWS resources like RDS or S3 buckets.
• Create custom inventory scripts for Azure or GCP if you operate multi‑cloud environments.
• Combine with Ansible Tower’s inventory management for a GUI‑based approach.

Remember to keep your ansible.cfg and inventory plugin files under version control, but never commit raw AWS credentials. Use the shared credentials file (~/.aws/credentials) or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) for secure access.