Making Airflow DAGs Resilient with Retries and SLAs

The problem: transient failures kill pipelines

By default an Airflow task instance fails the moment any upstream step raises an exception. The built‑in retry count is zero, so a momentary network glitch or a temporary database lock turns a whole DAG run red. Engineers often add custom retry logic, but that spreads boiler‑plate across every operator and makes DAGs harder to read.

Thesis

Airflow’s native retries, retry_delay and SLA fields give you a declarative way to absorb transient errors and to surface runs that exceed their expected window—no extra Python code required.

Understanding retries

A task can be told how many times to re‑execute and how long to wait between attempts. The parameters live on the task definition (or on the DAG to apply a default). When a task raises an exception, the scheduler re‑queues it up to retries times, respecting retry_delay. If all attempts fail, the task instance is marked failed and downstream tasks follow the normal trigger rules.

Configuring retries on a task

from datetime import timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator

def call_external_api():
    # raises on network error
    ...

with DAG(
    dag_id='api_ingest',
    schedule_interval='@hourly',
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:
    fetch = PythonOperator(
        task_id='fetch_data',
        python_callable=call_external_api,
        retries=3,
        retry_delay=timedelta(minutes=5),
    )

The snippet above tells Airflow to retry the fetch_data task three times, waiting five minutes between each attempt. The same values can be set on the DAG object (default_args={'retries': 3, 'retry_delay': timedelta(minutes=5)}) to apply them to every task unless overridden.

Using SLAs to catch missed windows

An SLA (Service Level Agreement) defines the latest time a task or DAG run should finish relative to its logical date. When the deadline passes, Airflow marks the run as sla_miss and can fire an alert via email, Slack, or a custom callback. SLAs are expressed as a timedelta on the task or DAG:

with DAG(
    dag_id='api_ingest',
    schedule_interval='@hourly',
    start_date=datetime(2024, 1, 1),
    catchup=False,
    sla=timedelta(hours=1),          # DAG‑level SLA
) as dag:
    fetch = PythonOperator(
        task_id='fetch_data',
        python_callable=call_external_api,
        retries=3,
        retry_delay=timedelta(minutes=5),
        sla=timedelta(minutes=30),   # task‑level SLA
    )

If the fetch_data task hasn’t succeeded within 30 minutes of the logical execution date, the UI shows an SLA breach and any configured sla_miss_callback runs.

Worked example: API call with retries and SLA

Assume an hourly DAG that pulls a JSON feed from a third‑party service. The service occasionally returns HTTP 5xx for a few seconds. The DAG below combines retries and a tight SLA so the team knows quickly when something is truly stuck.

from datetime import datetime, timedelta
import requests
from airflow import DAG
from airflow.operators.python import PythonOperator

def pull_feed():
    resp = requests.get('https://api.example.com/feed', timeout=10)
    resp.raise_for_status()          # raises on 4xx/5xx
    return resp.json()

def store_feed(**context):
    data = context['ti'].xcom_pull(task_ids='pull_feed')
    # write to warehouse …

with DAG(
    dag_id='hourly_feed',
    schedule_interval='@hourly',
    start_date=datetime(2024, 6, 1),
    catchup=False,
    default_args={
        'retries': 2,
        'retry_delay': timedelta(minutes=3),
    },
    sla=timedelta(minutes=45),
) as dag:
    pull = PythonOperator(
        task_id='pull_feed',
        python_callable=pull_feed,
        sla=timedelta(minutes=20),
    )
    store = PythonOperator(
        task_id='store_feed',
        python_callable=store_feed,
    )
    pull >> store

Result: a transient 5xx causes up to two retries spaced three minutes apart. If the feed still isn’t retrieved after 20 minutes, the task‑level SLA fires. If the whole DAG exceeds 45 minutes, the DAG‑level SLA fires. Both appear in the Airflow UI under **SLA Misses** and can trigger an alert.

Trade‑offs and limits

• Queue pressure: High retry counts on many concurrent tasks can saturate the executor queue, delaying unrelated work.
• Idempotency requirement: Retries re‑execute the exact same callable. If the task writes non‑idempotent side‑effects (e.g., appends to a log file without deduplication), repeated runs corrupt data.
• SLA timezone: SLA calculations use UTC. A DAG scheduled in a local timezone must have its start_date and schedule expressed in UTC, otherwise the breach window shifts unexpectedly.
• False alarms: An SLA that is tighter than the realistic worst‑case runtime generates noise; tune the window after observing actual runtimes.

How to verify the configuration

1. Open the DAG in the Airflow UI → **Graph View** → click a task → confirm retries and retry_delay values match the code.
2. Trigger a manual run, then inspect **Task Instance Details** → **SLA Miss** flag. The timestamp shown should be logical_date + sla (UTC).
3. Check the scheduler logs for lines like SLA miss for task … to ensure callbacks fire.

Next step

Add retries and retry_delay to the most flaky tasks in your current DAGs, then define a modest SLA (e.g., 1.5× the 95th‑percentile runtime). Monitor the **SLA Misses** page for a week; adjust the window or retry count based on real data. This small declarative change removes a whole class of custom retry code and gives you a built‑in reliability signal.