Balancing SLA Email Alerts vs Callback‑Driven External Notifications in Apache Airflow

Goal: Determine which alerting mechanism best reduces notification noise for SLA misses and task failures while maintaining scheduler stability.

Built‑in email alerts are enabled by default, can cause repeated emails during retries and backfills; callback‑driven external alerts require scheduler‑side callbacks that may delay scheduling if external API latency is high; Airflow version differences affect semantics.

Should we rely on default email SLA alerts with higher noise, or adopt callback‑driven external notifications despite added operational complexity? How can we verify scheduler impact of external callbacks without disrupting DAG execution? What criteria should define the acceptable noise threshold for SLA alerts?