Per-run checkpoint directories vs. atomic cloud storage writes for retry-safe tf.train.CheckpointManager
Scenario I am designing checkpointing for a multi-worker TensorFlow training job that must survive preemption and automatic retries. When a worker crashes and the job restarts, I need to guarantee that retries never leave duplicate or partially written checkpoint files in the target directory. Two documented options One documented approach is to give every t