Ansible Block Rescue for Safe Service Upgrades

When a task fails mid‑deployment

Imagine you are rolling out a new version of a web service. The playbook first stops the old instance, then installs the new package, and finally starts the updated service. If the installation step fails because of a missing dependency, the playbook aborts and the old service stays stopped, leaving users without access. You need a way to revert to the known‑good state and still notify the team, without writing external scripts.

What Ansible blocks give you

Ansible 2.0 introduced the block construct. A block groups a set of tasks; if any task inside the block fails, Ansible jumps to a rescue section where you can define compensatory actions. Regardless of whether the block succeeded, failed, or was rescued, an always section runs at the end. This mirrors try/catch/finally in many programming languages and keeps the error handling inside the playbook.

Worked example: safe service upgrade

The following playbook snippet shows a typical upgrade pattern. Replace myapp with the actual service name and adjust the package manager commands for your distribution.

- name: Upgrade myapp service
  hosts: webservers
  become: true
  tasks:
    - block:
        - name: Stop the old service
          ansible.builtin.service:
            name: myapp
            state: stopped
        - name: Install the new package
          ansible.builtin.yum:
            name: myapp
            state: latest
        - name: Start the upgraded service
          ansible.builtin.service:
            name: myapp
            state: started
      rescue:
        - name: Restart the old service as a fallback
          ansible.builtin.service:
            name: myapp
            state: started
        - name: Notify ops of the failure
          ansible.builtin.mail:
            host: smtp.example.com
            to: [contact removed]
            subject: "myapp upgrade failed on {{ inventory_hostname }}"
            body: "The upgrade playbook rescued a failure; the old service was restarted."
      always:
        - name: Record the outcome in a log file
          ansible.builtin.copy:
            dest: /var/log/myapp_upgrade_{{ inventory_hostname }}.log
            content: "Upgrade attempt finished at {{ ansible_date_time.iso8601 }} with result {{ ansible_job_result | default('unknown') }}"

Where to run: Save the snippet to upgrade.yml and execute ansible-playbook -i inventory upgrade.yml from a control node that has SSH access to the target hosts. The playbook requires become: true (root or sudo) to manage services and install packages.

Expected checks: If the yum task succeeds, you will see the service started and the always task create a log file. If the yum task fails (for example, because the repository is mis‑configured), Ansible skips the remaining block tasks, executes the rescue tasks (restarting the old service and sending the mail), and then runs the always task to write the log. You can verify the flow by adding -v to the command line and looking for the rescue and always task names in the output.

Practical way to check the result: After the run, inspect /var/log/myapp_upgrade_.log on each host to confirm the always task executed, and check the service status with systemctl status myapp (or the appropriate init command) to see whether the old service is running after a failure.

Trade‑offs and limitations

• Readability: Nesting blocks inside other blocks or mixing them with many rescue/always sections can make a playbook harder to follow. Keep each block focused on a single logical unit, such as “stop‑install‑start” for a service.
• Idempotency of rescue: The rescue section must be safe to run multiple times. In the example, restarting the old service is idempotent because state: started does nothing if the service is already running. Avoid actions that would accumulate side effects on retry, such as appending to a file without checking for duplicates.
• What rescue catches: Rescue only reacts to tasks that Ansible marks as failed (non‑zero return code, uncaught exception, etc.). It does not intercept external signals like SIGTERM or a process that crashes inside a started service unless that crash causes a subsequent task to fail.
• Always can mask errors: Because always runs even after a successful rescue, a failing always task will be reported but the original error may already have been handled. Treat always as cleanup, not as a place for critical logic that must succeed.

Actionable closing

When you need deterministic rollback or notification inside a playbook, reach for block/rescue/always before reaching for external scripts. Start small: wrap a single multi‑step operation in a block, add a rescue that restores the previous state, and use always for logging or notification. Verify the flow in a test environment with --check or -v, then promote to production. This keeps your automation self‑contained, visible, and easier to audit.