Docker volume restore after live backup: how to detect silent data inconsistency

Verifying Data Integrity After Live Restores

Because a tar-based backup of a running container captures a "crash-consistent" state at best, file hashes are insufficient. They confirm the file was copied, but not that the data inside the file is logically consistent. To detect silent inconsistency, you must move from bit-level verification to application-level validation.

Reliable Verification Methods

• Database Integrity Checks: Use native tools to scan for torn pages or broken indexes. For example, run PRAGMA integrity_check; for SQLite, CHECK TABLE for MySQL, or pg_checksums for PostgreSQL.
• Constraint Validation: Run queries to identify orphaned records or foreign key violations that may have occurred if a parent record was backed up but its child record was written just after the tar process passed that directory.
• Business Logic Smoke Tests: Execute a set of read-only reports or data aggregations on the restored volume and compare the results against a known baseline from the source.
• Filesystem Metadata Scan: If you have access to the backing device, run fsck -n (read-only mode) to ensure the filesystem structure itself isn't corrupted, though this will not catch application-level logical errors.

Atomicity and Driver Guarantees

There are no documented Docker Engine options or flags that provide atomicity for the local volume driver using a tar-based approach. The local driver simply maps a host directory to the container; it does not implement snapshotting or copy-on-write mechanisms for backups.

To achieve a consistent backup without stopping the container, you must use one of the following:

• Application Quiescence: Trigger a "flush to disk" or "backup mode" command within the app before running the tar.
• Host-Level Freezing: Use fsfreeze on the host (if using ext4/xfs) to momentarily pause I/O, though this requires root access and affects all volumes on that mount.
• External Snapshotting: Use a volume driver that supports CSI snapshots (e.g., AWS EBS or Azure Disk), which operates at the block level rather than the file level.

Delegating Consistency to Application Tools

For databases or write-heavy applications, consistency should be delegated entirely to application-level dump tools (e.g., pg_dump, mysqldump). Volume-level backups of active databases are inherently risky because they capture the data files in a state equivalent to a sudden power loss.

A robust strategy typically involves a hybrid approach: use application dumps for the database and tar for static assets/uploads, ensuring the database dump is the single source of truth for relational data.

Diagnostic Requirement

To refine this recommendation: Which specific database or storage engine is the application using? (The verification commands differ significantly between a document store like MongoDB and a relational store like PostgreSQL).