GitLab backup restore: are LFS objects in uploads actually recovered?
0 reputation · 28 Feb 2023, 23:21 UTC
0 reputation · 28 Feb 2023, 23:21 UTC
GitLab's built-in gitlab:backup:create task bundles the database, repositories, uploads, and CI artifacts into one timestamped tarball, and gitlab:backup:restore brings the instance back from it. The stated goal here is a complete disaster-recovery restore where every project's Git LFS files are usable again without manual re-upload.
The uncertainty is around LFS specifically. The backup does include the uploads directory, which is where LFS objects live on a local-disk setup, but the restore documentation does not clearly state that those objects are extracted and re-linked so LFS pointers resolve again. There is also the complication that instances using external object storage (for example S3) only get metadata in the tarball, with the actual objects living outside the backup entirely. Add in the same-major-version restore requirement and the fact that runner registration data is excluded, and it is hard to know what "fully recovered" really means.
Assume a recent GitLab version on Linux package (Omnibus) installation with local storage. Verification ideas considered include comparing gitlab:backup:checksum output and spot-checking LFS files through the UI or API after restore.
uploads so existing LFS pointers resolve, or is a separate step required?32775 reputation · 01 Mar 2023, 08:23 UTC
Yes — on an Omnibus install with local storage, a standard gitlab:backup:restore recovers LFS objects, but the premise in the question needs one correction: LFS objects are not stored in uploads. They are a separate backup component, lfs-objects, archived alongside uploads, artifacts, repositories, and the database dump inside the same timestamped tarball. The restore task extracts each component back to its configured path, so existing LFS pointers resolve again without manual re-upload.
Confirmed behavior for recent Omnibus versions:
/var/opt/gitlab/gitlab-rails/shared/lfs-objects), not under uploads. Uploads holds attachments, avatars, and similar files.gitlab:backup:create includes lfs-objects as its own archive inside the tarball; restore unpacks it to the configured location.gitlab-secrets.json must be restored separately or encrypted database values cannot be decrypted.Assumed: a recent GitLab version on Omnibus with default local storage. Exact component lists and available rake tasks vary by version, so treat version-specific details as needing a check against your installed version's documentation.
If LFS is configured to use S3-compatible object storage, the tarball contains only database metadata — the object bytes were never in the backup. "Fully recovered" then means: restore the backup, configure the instance to point at the same bucket, and the pointers resolve against the still-existing objects. If the bucket was deleted or lifecycle rules expired the objects, no restore can bring them back; object storage needs its own backup/replication strategy (versioning, cross-region replication) independent of gitlab:backup:create.
There is no official rake task that verifies LFS content integrity end to end. Practical checks:
lfs-objects archive is present.git lfs pull or git lfs fsck to confirm pointers resolve and content downloads intact.gitlab:backup:checksum can generate component checksums for backup-level comparison — useful, but it validates the archive, not LFS semantics.If your restore test shows LFS downloads failing while repositories load fine, the one diagnostic detail that changes the recommendation is your gitlab.rb LFS storage setting (local path vs. object storage) — that determines whether you are debugging an extraction problem or a bucket-configuration problem.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.