Task Retry Idempotency and Side Effect Duplication

Answer

The deterministic idempotency key that remains unchanged across automated retries and manual task clears is a combination of the DAG run identifier and the task identifier:

{{ dag_run.run_id }}_{{ task.task_id }}

Alternatively, you can use the logical date:

{{ dag.dag_id }}_{{ execution_date }}_{{ task.task_id }}

Why this works

• dag_run.run_id (or execution_date) is fixed for the lifetime of a DAG run and is not altered when a task is cleared from the UI.
• task.task_id is static per task.
• Airflow’s try_number increments on each automatic retry and resets to 1 after a manual clear, but it is not part of the key, so the key stays stable.

Steps to implement idempotency

1. In your operator (e.g., PythonOperator or custom operator), retrieve the context and build the key:

def execute(self, context):
    run_id = context['dag_run'].run_id
    task_id = context['task'].task_id
    idempotency_key = f"{run_id}_{task_id}"
    # pass the key and try_number to the external system
    try_number = context['ti'].try_number
    # Example: include as HTTP header or DB column
    call_external_api(idempotency_key, try_number, ...)

1. In the external system (database, API, message queue), store the maximum try_number seen for each idempotency_key. On receipt of a new attempt:

• If the incoming try_number is greater than the stored maximum, treat it as a new automated retry and process the side‑effect (using an upsert or conditional insert).
• If the incoming try_number is less than or equal to the stored maximum, the attempt is a manual re‑run; skip the side‑effect or perform a no‑op.

Confirmation of behavior

Airflow increments try_number for each automatic retry and leaves dag_run.run_id unchanged. A manual task clear via the UI resets try_number to 1 while keeping the same run_id, execution_date, dag_id, and task_id. Therefore the key built from run_id and task_id (or dag_id, execution_date, task_id) is stable across both automated retries and manual clears.