Remote Logging Configuration Transition for Ephemeral Worker Environments

Answer

Airflow does not automatically clean up or migrate logs when you change the remote logging provider. The remote_logging flag only tells the worker to write to whatever storage is configured at runtime; the metadata database still holds a log_location_prefix that points to the original bucket or path. When you point Airflow to a new backend, the UI will continue to look for logs using the old prefix unless you update that value.

Legacy Log Cleanup

• Airflow never deletes remote log files on its own. Log rotation is handled by the worker process – it creates a new file for each TaskInstance run, but never removes old files.
• If you change the provider (e.g., from S3 to GCS), the old bucket remains untouched. You must run a cleanup job or rely on the cloud provider’s lifecycle rules to purge the stale logs.
• Airflow’s remote_log_conn_id and log_location_prefix are stored in the metadata DB. Updating the configuration alone does not alter those values; you need to manually edit the DB or use the airflow db upgrade hooks if you migrate the schema.

UI Retrieval After Migration

• If the UI still references the old provider in the metadata, it will try to fetch logs from that storage. If the worker no longer has permission or the bucket has been deleted, the UI will show an error message or an empty log view.
• When the log_location_prefix is updated to the new path, the UI will correctly locate logs in the new backend, assuming the webserver has the same credentials and provider libraries.
• Because the log path is derived from the DAG ID, task ID, execution date, and try number, you can manually navigate to the new location in the storage console to verify that logs exist.

Practical Steps to Transition

1. Set remote_logging = True and configure remote_log_conn_id for the new backend.
2. Update log_location_prefix in airflow.cfg or via the environment variable AIRFLOW__CORE__LOG_LOCATION_PREFIX to match the new bucket/path.
3. Restart the webserver and scheduler so they pick up the new configuration.
4. Verify that new task runs write to the new backend by checking a recent log file.
5. Manually delete or set a lifecycle policy on the old bucket to remove legacy logs.

Need One More Detail?

To fine‑tune the cleanup recommendation, could you confirm whether the log_location_prefix was left unchanged when you switched providers? If the prefix still points to the old bucket, you’ll need to update it before the UI can retrieve the new logs.