LocalExecutor vs CeleryExecutor for filesystem-dependent tasks

Question

When transitioning from a local development environment to a production cluster, DAGs that rely on shared local filesystems often fail. The LocalExecutor executes tasks as subprocesses on the scheduler node, allowing tasks to read and write files from the same /tmp directory seamlessly.

However, moving to CeleryExecutor distributes tasks across remote workers via a broker. This architecture ensures that any file written to a worker's local disk is invisible to downstream tasks running on different nodes, resulting in FileNotFoundError.

This creates a conflict between maintaining the simplicity of a single-node setup and the necessity of distributed execution for high-volume workloads.

Is it more maintainable to refactor DAGs to use external object storage for intermediate data, or are there specific configurations to ensure a shared filesystem across Celery workers nodes?

LocalExecutor vs CeleryExecutor for filesystem-dependent tasks

When transitioning from a local development environment to a production cluster, DAGs that rely on shared local filesystems often fail. The LocalExecutor executes tasks as subprocesses on the scheduler node, allowing tasks to read and write files from the same /tmp directory seamlessly.

0 answers

Your perspective belongs here.

0 question comments