Handling Long-Running Tasks in Ansible with Async

The Timeout Problem in Infrastructure Automation

When managing infrastructure, you occasionally encounter tasks that exceed the default SSH timeout limits. Whether it is a large database migration, a massive package update, or a complex cloud resource deployment, these "long-haul" tasks often cause Ansible to drop the connection, leaving you unsure if the process is still running on the remote host or if it crashed mid-execution.

The solution is to decouple the task execution from the Ansible connection using the async and poll parameters. This allows you to fire off a command and either check back later or move on to other tasks entirely, preventing the "SSH timeout" failure while maintaining control over the process.

How Async and Poll Interact

To implement asynchronous execution, you must define two specific parameters in your task:

• async: The maximum time (in seconds) the task is allowed to run before Ansible considers it failed.
• poll: How often Ansible should check the remote host for the task's completion.

Setting poll: 0 is the critical move for "fire-and-forget" operations. When poll is 0, Ansible starts the task and immediately moves to the next play without waiting for a result. If poll is set to a positive integer (e.g., poll: 5), Ansible will wait for the task to finish but will check the status every 5 seconds, which helps keep the SSH session alive during long waits.

Worked Example: Backgrounding a Long Process

In this scenario, we simulate a long-running process (like a system backup) that we want to start and then verify later in the playbook. This example assumes Ansible 2.9+ and a Linux target.

- name: Handle long running backup
  hosts: db_servers
  become: yes
  tasks:
    - name: Start a long-running backup process
      ansible.builtin.shell: "sleep 60 && echo 'Backup Complete'"
      async: 120
      poll: 0
      register: backup_job

    - name: Perform other unrelated configuration tasks
      ansible.builtin.debug:
        msg: "I am doing other work while the backup runs in the background!"

    - name: Wait for the backup to finish
      ansible.builtin.async_status:
        jid: "{{ backup_job.ansible_job_id }}"
      register: job_result
      until: job_result.finished
      retries: 30
      delay: 2

    - name: Report backup status
      ansible.builtin.debug:
        var: job_result.stdout

Execution Details: Run this via ansible-playbook site.yml. The async_status module uses the jid (Job ID) registered from the first task to track the specific process on the remote host. The until loop ensures the playbook pauses only when it absolutely needs the result of the backup to proceed.

Trade-offs and Operational Risks

While async solves timeout issues, it introduces complexity regarding state and idempotency. Because the playbook can move past a task before it finishes, you risk race conditions. For example, if Task A starts a service asynchronously and Task B attempts to configure that service, Task B may fail because the service is not yet fully initialized.

Furthermore, async tasks can become orphaned. If the Ansible controller crashes or the playbook is manually stopped while a poll: 0 task is running, the process continues on the remote server. Unlike standard tasks, there is no automatic rollback for an async process once it has been handed off to the remote system's process manager.

Verification and Testing

To verify that your async implementation is working as intended, check the following:

1. Timing: Observe the playbook output. The task with poll: 0 should return a status almost instantly, regardless of the actual command duration.
2. Job ID: Ensure the ansible_job_id is present in the registered variable; without this ID, async_status cannot track the task.
3. Exit Codes: Check the job_result.rc value. A 0 indicates success, while any other number indicates the background process failed, even if the initial "fire" command succeeded.