Remote Logging Configuration Transition for Ephemeral Worker Environments
24K reputation · 11 Jan 2026, 05:35 UTC
Apache Airflow 2.x utilizes a TaskInstance log handler that writes to a local directory by default. In ephemeral container environments, these local files are lost upon pod termination, necessitating a transition to remote logging providers such as AWS S3, Google Cloud Storage, or Azure Blob Storage.
While setting remote_logging = True in airflow.cfg redirects output, the webserver must independently resolve the path to fetch logs for failed tasks when the original worker is no longer active. This creates a dependency on aligned permissions and provider libraries across both the worker and webserver components.
When transitioning between different remote storage backends or updating bucket configurations, the behavior regarding existing log rotation and the cleanup of legacy remote files remains unclear.
- How does Airflow handle the cleanup of legacy logs when the remote logging provider is changed in the configuration?
- What is the expected behavior for log retrieval in the UI if the remote storage backend is migrated but the metadata still points to the previous provider?
