os.replace vs. tempfile.NamedTemporaryFile for idempotent file writes

Direct answer

os.replace atomicity on NAS: No. os.replace is atomic on local POSIX and Windows filesystems when source and target are on the same mount. On network-attached storage, NFS, SMB/CIFS, and many cloud object gateways, rename can be implemented as copy-delete or may not be atomic across clients. Atomicity is not guaranteed and cannot be relied on for idempotency on NAS.

Safe NamedTemporaryFile cleanup without deleting the target: Yes. Use tempfile.NamedTemporaryFile(delete=False), keep the returned temp path, call os.replace(temp_path, target_path), and only delete the temp path if it still exists and is different from the target. Never delete using a reused variable.

Confirmed facts

• Python 3.11+ os.replace(src, dst) maps to the OS rename syscall. On POSIX and Windows it replaces dst atomically if src and dst are on the same filesystem.
• tempfile.NamedTemporaryFile(delete=False) creates a file that persists after close. The name is returned and must be managed manually.
• If the process crashes after writing the temp file but before os.replace, the target is unchanged and an orphan temp remains.

Likely explanation, not guaranteed

• On NAS or cross-mount renames, the operation may fall back to copy-delete, exposing a window where readers see partial data or the replace fails partway.
• Some NFS configurations with weak cache coherence can make rename appear atomic locally but not cluster-wide.
• On filesystems without atomic rename (e.g. FAT, some NFS configurations), consider write-copy-rename with a journal or a distributed lock to achieve true idempotency across retries.

Minimal idempotent write pattern for this case

1. Create temp in the same directory as target with a unique prefix.
2. Write and fsync data, close file.
3. os.replace(temp_path, target_path) inside try/except.
4. In finally, if os.path.exists(temp_path) and temp_path != target_path, unlink temp_path.
5. On retry, remove any leftover temp files from previous attempts before starting.

import os
import tempfile

def write_idempotent(target_path, data):
    dir_name = os.path.dirname(os.path.abspath(target_path))
    tmp = tempfile.NamedTemporaryFile(
        mode='wb',
        dir=dir_name,
        delete=False,
        prefix='.tmp-',
        suffix='.dat'
    )
    temp_path = tmp.name
    try:
        with tmp:
            tmp.write(data)
            tmp.flush()
            os.fsync(tmp.fileno())
        os.replace(temp_path, target_path)
    except Exception:
        raise
    finally:
        if os.path.exists(temp_path) and os.path.abspath(temp_path) != os.path.abspath(target_path):
            try:
                os.unlink(temp_path)
            except FileNotFoundError:
                pass

Caution: manual cleanup can accidentally delete the target if the path variable is reused. Always compare absolute paths before unlinking.

One missing diagnostic detail that changes the recommendation: what filesystem and mount point are used for target and temp, and what NAS protocol/version is in play. If temp and target are not on the same POSIX filesystem, atomic replace cannot be relied on and an additional journal or lock is needed.