Reproducible Builds on AlmaLinux with the AlmaLinux Build System (ABS)

Why Reproducible Builds Matter on AlmaLinux

AlmaLinux’s commitment to binary compatibility with Red Hat Enterprise Linux (RHEL) is built on the AlmaLinux Build System (ABS). ABS lets you rebuild any RHEL‑derived package from source, producing an identical binary that is signed with the AlmaLinux GPG key. This guarantees that your custom builds are trustworthy, auditable, and fully compliant with the distribution’s security policies.

1. Requirements Checklist

• ABS package – Install with dnf install abs. Verify with rpm -q abs.
• Mock or rpmbuild – ABS relies on mock to create isolated chroots. Ensure mock is present: dnf install mock.
• Source RPMs (SRPMs) – Obtain from the AlmaLinux Vault or debuginfo repositories. SRPMs are signed with the AlmaLinux GPG key.
• GPG key – The AlmaLinux key should be imported into your keyring. Check with rpm -qi gpg-pubkey for a key ending in almalinux.org.
• Disk space & CPU – Large packages (e.g., kernel, Mesa) can consume >10 GB and several hours of CPU time.

2. Minimal Design: A Single Mock Chroot Build

The smallest, most reliable workflow uses a dedicated mock configuration that mirrors the target RHEL release. For example, to rebuild httpd-2.4.46-1.el8 for AlmaLinux 8 on an x86_64 host, follow:

# 1. Locate the SRPM
srpm_path="/srv/almalinux/vault/8.6/os/x86_64/SRPMS/httpd-2.4.46-1.el8.src.rpm"

# 2. Rebuild in the appropriate mock chroot
mock -r almalinux-8-x86_64 rebuild $srpm_path

# 3. Verify the output RPM
output_dir="~/rpmbuild/RPMS/x86_64"
for rpm in $output_dir/*.rpm; do
    rpm -K $rpm
    rpm -qp --qf "%{NVR}\n" $rpm
done

Key points:

• The almalinux-8-x86_64.cfg mock file pulls the exact build dependencies from the AlmaLinux mirrors.
• ABS does not modify the spec file; it uses the upstream spec unchanged, preserving binary compatibility.
• The build runs as an unprivileged user inside the chroot, ensuring isolation.

3. Trust and Data Boundaries

Component	Trust Boundary	Verification Step
Source RPM	Signed by AlmaLinux GPG key	rpm -K httpd-2.4.46-1.el8.src.rpm
Mock Chroot	Runs as non‑root, isolated environment	Inspect /etc/mock/almalinux-8-x86_64.cfg for root:mock
Binary RPM	Signed with AlmaLinux key	rpm -K httpd-2.4.46-1.el8.x86_64.rpm
Build Dependencies	Fetched from AlmaLinux mirrors	Check mock.log for Downloading entries

4. Operational Checks After Build

1. Signature Verification – rpm -K *.rpm should report pgp: OK for all packages.
2. Filelist Consistency – Compare the filelist of the rebuilt package with the upstream binary. Use rpm -qlp rebuilt.rpm vs. rpm -qlp upstream.rpm and diff.
3. Test Suite Execution – If the SRPM contains a tests/ directory, run mock -r ... --run-tests or manually invoke make test inside the chroot.
4. NVR Matching – The Name-Version-Release string should match the upstream NVR unless you intentionally modify the Release field.
5. Binary Size & SHA256 – For high assurance, compute sha256sum of the binary and compare with the upstream checksum if available.

5. Failure Modes and When to Re‑design

• GPG Key Rotation – If AlmaLinux rotates the signing key, previously built binaries will fail rpm -K. Update your keyring and rebuild.
• Missing or Corrupt SRPM – A download error or corrupted SRPM will abort the build. Verify rpm -q --qf "%{SIGPGP:pgpsig}\n" $srpm before building.
• Mock Misconfiguration – Incorrect mock config (e.g., wrong repo URLs) leads to missing dependencies. Check mock.log for “No such file or directory” errors.
• Dependency Changes – Upstream patches that add new build dependencies require updating the mock base image. Use mock --init to refresh.
• Custom Patches or Spec Modifications – Applying patches breaks the binary compatibility guarantee. If you must patch, document the change and consider maintaining a separate build branch.
• Resource Exhaustion – Large builds may exceed disk or CPU limits. Monitor df -h and CPU usage during the build.

When to Shift to a More Complex Design

If you need to:

• Patch upstream sources for security or feature reasons.
• Integrate with CI/CD pipelines that require reproducible builds across multiple architectures.
• Produce packages for custom target environments (e.g., minimal or embedded).

In those cases, consider:

• Using ABS with --patch to apply curated patches.
• Creating multiple mock configurations for each target architecture.
• Automating the build with buildfarm or a containerized mock runner.

6. Practical Verification Checklist

• Confirm ABS installation: rpm -q abs returns a version.
• Verify mock config exists: ls /etc/mock/almalinux-8-x86_64.cfg.
• Test a quick build (e.g., bash or coreutils) to ensure the environment works.
• Run rpm -K on the resulting binary; expect pgp: OK.
• Use rpm -V to verify file integrity against the spec.

7. Summary

AlmaLinux’s ABS provides a straightforward path to reproducible, RHEL‑compatible binaries. By adhering to the minimal design—using the official mock configuration, unmodified spec files, and signed SRPMs—you maintain the distribution’s security guarantees while enjoying the flexibility of source builds. Operational checks and a clear understanding of failure modes ensure that your builds remain trustworthy and auditable.