Per-run checkpoint directories vs. atomic cloud storage writes for retry-safe tf.train.CheckpointManager

Short answer

tf.train.CheckpointManager does not guarantee atomic checkpoint creation on any filesystem. It handles retention (max_to_keep) and checkpoint discovery within one directory; the atomicity story comes entirely from the underlying tf.io.gfile backend, and even there it is per-object, not per-checkpoint. For retry-heavy distributed jobs, the safer documented pattern is the per-run unique directory, ideally on object storage, with only the chief worker saving.

Why CheckpointManager alone cannot promise atomicity

A TensorFlow 2.x checkpoint is not one file. A save produces a .index file, one or more sharded .data files, and an update to the checkpoint state proto that records the latest prefix. A crash anywhere in that sequence can leave a state file pointing at a prefix whose data files were never fully written. CheckpointManager has no transactional commit step that would make this multi-file set all-or-nothing, on any backend.

What the storage layer adds is narrower than it looks:

• GCS (gs://): object writes are effectively all-or-nothing at the object level — an object becomes visible only on successful completion. That eliminates torn individual files, but a crash between objects still leaves an incomplete checkpoint set.
• S3 (s3://): support comes via filesystem plugins whose semantics vary by plugin and version. Multipart upload completion gives per-object atomicity, but there is no multi-object transaction, and rename-based patterns are unreliable.
• Local POSIX: rename is atomic per file, which helps single files but again not the whole set.

So the answer to your first question: atomicity is a property of the backend, and only at single-file granularity — never a CheckpointManager-level guarantee.

The safer pattern for preemptible, retried jobs

Give each run (or each attempt) its own directory, e.g. gs://bucket/job/<run-id>/ckpt. The decisive property is not atomicity but isolation: a retried run writes into a fresh path, so a partially written checkpoint from a crashed attempt can never be discovered by tf.train.latest_checkpoint in the new run's directory. This holds regardless of backend semantics, which is exactly what you want when you cannot confirm those semantics across filesystems and versions.

Two requirements make this work:

1. Chief-only saving. In a multi-worker setup, only the chief should call CheckpointManager.save. Concurrent saves from several workers into one directory are not a safe supported pattern on any backend, atomic or not.
2. An external restore pointer. Since each run has a new directory, record the last known-good checkpoint path outside the run directory (a small marker object, job metadata, or your orchestrator's state) so a retry restores from the last fully completed attempt rather than scanning for directories.

Trade-offs to weigh

Write latency is roughly comparable — both write the same bytes. The real cost difference is cleanup overhead versus correctness risk, and for preemptible jobs correctness should win.

Verify on your actual stack

Backend behavior is version- and plugin-dependent, so treat the above as a design default, not a substitute for testing:

• Kill a worker mid-save on your target backend, then check whether tf.train.latest_checkpoint returns a prefix whose data files are missing or truncated.
• Run a preemption-and-retry loop in staging and assert the restored state matches the last fully completed step, with no duplicate checkpoint prefixes.
• Confirm in your strategy configuration that only the chief saves.
• Check the tf.io.gfile backend docs for your installed TensorFlow version's write/rename semantics for your scheme.

One caveat worth flagging: exact GCS/S3 filesystem behavior differs across TensorFlow 2.x releases and plugin versions, so the per-object atomicity claims above should be re-verified against your pinned versions before you rely on them.