Optimizing Immutable Ubuntu AMIs with Packer’s Ansible Provisioner

Problem Statement

When creating production‑ready Ubuntu Amazon Machine Images (AMIs) with Packer, teams often struggle to keep the image size small, enforce a minimal attack surface during build time, and guarantee that the configuration is repeatable and auditable. The combination of Packer’s amazon-ebs builder and the Ansible provisioner offers a declarative path to address these concerns, but the details of SSH handling, inventory management, and artifact cleanup are frequently overlooked.

Requirements

• Ubuntu 22.04 LTS builder instance (or any supported Ubuntu version).
• Packer 1.9+ and Ansible 2.12+ installed on the host that runs the build.
• IAM role with ec2:CreateImage and s3:PutObject permissions for the builder.
• A Git‑controlled playbook repository that contains idempotent tasks and a minimal set of packages.
• SSH key pair that is generated for the build only and never persisted in the image.

Smallest Suitable Design

The goal is to produce an AMI that contains only the runtime artifacts required by the application, no build‑time tools, and no residual temporary files.

1. Builder instance: Spin up an Ubuntu 22.04 EBS‑optimized instance with a temporary key pair.
2. SSH configuration: Pass --ansible-ssh-extra-args \"-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null\" to eliminate host‑key checks and prevent the key from being cached.
3. Ansible connection: Use ansible_connection=ssh with the temporary key, or ansible_connection=local if the playbook runs on the builder itself.
4. Playbook design: Keep tasks idempotent, avoid shell or command modules that create large temp files, and explicitly clean up with file: state=absent.
5. Artifact pruning: Add a final file task that removes /var/lib/apt/lists/* and /tmp/* before the image snapshot.
6. Image commit: Let Packer snapshot the instance; the resulting AMI is immutable because the instance is terminated immediately afterward.

Trust & Data Boundaries

• The temporary SSH key is generated on the build host and injected into the amazon-ebs builder only for the duration of the build. It is never written to the filesystem of the instance or the resulting AMI.
• All configuration data resides in the version‑controlled playbook repository. Packer pulls the repository at build time, ensuring that the exact commit used for the image is auditable.
• Network traffic between the builder and the instance is confined to the AWS VPC. The builder’s security group allows only outbound SSH to the instance, and inbound SSH is closed after the build.

Operational Checks

1. Debug build:

   packer build -only=amazon-ebs -debug template.json
   

   Check the generated packer.log for sections tagged \"ansible\" to confirm that the playbook ran and that the expected changes were applied.
2. Check mode validation:

   packer build -only=amazon-ebs -debug -var 'ansible_check_mode=true' template.json
   

   The log should report \"0 changes\" and exit without modifying the instance state.
3. Post‑build verification: After the AMI is created, launch an instance and run:

   ansible --version
   ansible -i localhost, -m ping
   

   This confirms that the Ansible binaries and modules are present and that the system state matches the playbook.
4. Size comparison: Use the AWS console or aws ec2 describe-images --image-ids <ami-id> --query \"Images[0].ImageSizeInBytes\" before and after adding the provisioner to quantify the impact on image size.

Failure Modes & Conditions that Change the Design

• Missing Python runtime: The Ansible provisioner requires a functional Python environment on the builder. If python3 or required libraries are absent, the provisioner fails before any image changes are committed. Solution: include a pre‑build script that installs Python and the python3-apt package.
• Kernel module drift: Tasks that load or modify kernel modules may not be fully captured in the snapshot. If the application depends on a custom module, consider using modprobe in a shell task and ensuring the module is present in the final image.
• Large temporary files: Ansible tasks that download archives or compile code can leave large artifacts in /tmp or package caches. Failing to prune these inflates the AMI size. Always add a cleanup task at the end of the playbook.
• Persistent SSH keys: Accidentally leaving the temporary key in /root/.ssh/authorized_keys or the instance’s filesystem exposes the image to unauthorized access. Verify the key is removed before snapshotting or use the ssh_keypair_name option to let Packer clean up automatically.
• Network latency or timeouts: The builder may time out while waiting for SSH. Increase the ssh_timeout in the builder configuration or use ssh_wait_timeout to give the instance more time to become reachable.

Practical Example

Below is a minimal template.json that demonstrates the key concepts. Replace placeholders with your own values.

{
  "builders": [
    {
      "type": "amazon-ebs",
      "region": "us-east-1",
      "source_ami": "ami-0abcdef1234567890",
      "instance_type": "t3.micro",
      "ssh_username": "ubuntu",
      "ssh_keypair_name": "packer-temp-key",
      "ssh_private_key_file": "path/to/temp/key.pem",
      "ami_name": "ubuntu-22.04-ansible-{{timestamp}}",
      "ami_description": "Immutable AMI built with Packer and Ansible",
      "run_tags": {"Project": "Infra"},
      "tags": {"OS": "Ubuntu22.04", "Purpose": "Immutable"},
      "temporary_security_group_rules": [
        {"type": "ingress", "protocol": "tcp", "port": 22, "cidr_blocks": ["0.0.0.0/0"]}
      ],
      "ssh_timeout": "5m"
    }
  ],
  "provisioners": [
    {
      "type": "ansible",
      "playbook_file": "playbooks/setup.yml",
      "extra_arguments": ["--extra-vars", "ansible_check_mode={{ ansible_check_mode }}"],
      "ansible_ssh_extra_args": "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null",
      "ansible_connection": "ssh"
    }
  ],
  "variables": {
    "ansible_check_mode": {"default": false}
  }
}

Playbook example (setup.yml):

- hosts: all
  become: true
  vars:
    packages:
      - htop
      - git
  tasks:
    - name: Update apt cache
      apt: update_cache=yes cache_valid_time=3600
    - name: Install base packages
      apt: name={{ packages }} state=present
    - name: Clean apt caches
      file:
        path: /var/lib/apt/lists/*
        state: absent
    - name: Remove temporary files
      file:
        path: /tmp/*
        state: absent

Run the build with:

packer build -var 'ansible_check_mode=true' template.json

When ansible_check_mode is true, Ansible will run in check mode and report zero changes, allowing you to validate the playbook without altering the instance.

Limitations & Future Enhancements

• The Ansible provisioner does not automatically remove temporary SSH keys from the instance’s authorized_keys file. Consider adding a cleanup task or using the ssh_keypair_name feature to let Packer manage key lifecycle.
• Kernel module changes may not be fully captured; if your application requires custom modules, you may need a post‑build script that re‑applies them.
• Large dependency downloads (e.g., language runtimes) can still inflate the final AMI size. Use apt-get purge or --no-install-recommends to keep the image lean.

By following this architecture note, teams can produce secure, minimal, and repeatable Ubuntu AMIs suitable for production workloads on AWS.