RHSM Registration and Repository Drift: A Minimal Architecture for Stable RHEL Package Management

The Problem: Silent Repository Drift After Pool Rebalancing

\n

You register a RHEL system with subscription-manager register, attach a subscription pool, and dnf install works. Months later, a pool rebalance on the Red Hat side shifts entitlements. The local /etc/yum.repos.d/redhat.repo file still points to the old repository URLs, but the CDN now returns 403 for those channels. dnf reports missing packages or dependency errors with no obvious cause. The registration looks current; the subscription shows attached. The failure is a trust boundary violation: the local repository metadata no longer reflects the authoritative subscription state.

\n\n

Requirements

\n

\n
• Systems must retrieve packages only from repositories matching their current subscription entitlements.
\n
• Repository configuration must self-correct when pool membership changes, without manual file edits.
\n
• Operators must verify alignment between subscription state and enabled repositories in a single command.
\n
• Major version upgrades (e.g., RHEL 8 → 9) must migrate subscriptions and repositories atomically.
\n

\n\n

Smallest Suitable Design: RHSM-Managed Repositories Only

\n

Do not maintain static .repo files. Let subscription-manager own /etc/yum.repos.d/redhat.repo entirely. The daemon rewrites this file on every subscription-manager refresh and on rhsmcertd certificate renewal (default 4-hour interval). This makes the repository list a derived artifact of the active subscription pools, not a configuration input.

\n

Disable any third-party or manually created .repo files that point to Red Hat CDN base URLs. If you need custom repositories (e.g., internal mirrors), place them in separate files under /etc/yum.repos.d/ with distinct names and enabled=0 by default, activating them only via dnf --enablerepo= when required.

\n\n

Trust and Data Boundaries

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Boundary	Authority	Data Flow	Verification Point
Subscription Pool → Local System	Red Hat Customer Portal / Satellite	Entitlement certificates (X.509) downloaded to /etc/pki/entitlement/	subscription-manager identity shows current consumer UUID and installed certificates
Entitlement Certificate → Repository List	subscription-manager (local)	Certificate extensions enumerate permitted product IDs and content sets; rhsmcertd regenerates redhat.repo	subscription-manager repos --list-enabled matches dnf repolist enabled
Repository Metadata → DNF	Red Hat CDN (or Satellite)	Signed repodata (repomd.xml, primary.xml.gz) fetched over HTTPS with certificate authentication	dnf repoinfo shows GPG key and SSL client cert in use

\n

The critical boundary is the entitlement certificate. It is the single source of truth. If the certificate expires or is revoked, rhsmcertd disables the associated repositories by setting enabled=0 in redhat.repo. DNF then ignores them silently.

\n\n

Operational Checks

\n

Daily: Automated Alignment Verification

\n

Add a cron job or systemd timer running:

\n

#!/bin/bash\n# /usr/local/bin/rhsm-repo-check\nset -euo pipefail\n\n# 1. Verify subscription-manager can reach the portal\nsubscription-manager status --json | jq -e '.overall_status == \"Current\"' || exit 1\n\n# 2. Compare enabled repos from RHSM vs DNF\nrhsm_repos=$(subscription-manager repos --list-enabled --json | jq -r '.[].id' | sort)\ndnf_repos=$(dnf repolist --enabled --json | jq -r '.repos[].id' | sort)\n\nif ! diff -u <(echo \"$rhsm_repos\") <(echo \"$dnf_repos\") >/dev/null; then\n    logger -t rhsm-drift \"Repository mismatch detected\"\n    subscription-manager refresh\n    exit 1\nfi\n

\n

This catches drift within the rhsmcertd refresh window. The subscription-manager refresh forces immediate certificate re-evaluation and repository regeneration.

\n\n

On-Demand: Full State Dump

\n

# Run as root on the RHEL host\nsubscription-manager status\nsubscription-manager list --consumed --pool-only\nsubscription-manager repos --list-enabled\ndnf repolist all\ncat /var/log/rhsm/rhsm.log | tail -50\n

\n

Cross-reference the pool IDs from list --consumed with the repository IDs in repos --list-enabled. Each pool maps to one or more content sets (e.g., rhel-9-for-x86_64-baseos-rpms). A missing mapping indicates either an unattached pool or a certificate issue.

\n\n

Failure Modes

\n

1. Pool Rebalance Without Re-attachment

\n

Red Hat moves a subscription from one pool to another (common during contract renewals). The old pool ID remains in /etc/pki/entitlement/ until rhsmcertd renews (up to 4 hours). During this window, dnf fails with Error: Failed to download metadata for repo 'rhel-9-for-x86_64-baseos-rpms': 403 Forbidden.

\n

Mitigation: Reduce rhsmcertd interval in /etc/rhsm/rhsm.conf:

\n

[rhsmcertd]\ncertFrequency = 60  # minutes, default 240\n

\n

Or trigger manually after known contract changes: subscription-manager refresh.

\n\n

2. Stale Repository Files After Major Upgrade

\n

Upgrading RHEL 8 → 9 via leapp migrates the subscription but leaves redhat.repo with RHEL 8 repository IDs. The entitlement certificate now authorizes RHEL 9 content sets, but the old file still declares rhel-8-for-x86_64-baseos-rpms. DNF sees both, prefers the higher-versioned (but unauthorized) RHEL 8 repo, and fails dependency resolution.

\n

Fix: After leapp reboot, run:

\n

subscription-manager remove --all\nsubscription-manager register --activationkey= --org=\nsubscription-manager refresh\n

\n

This forces a clean certificate fetch and repository regeneration for the new major version.

\n\n

3. Satellite-Managed Systems Bypassing CDN

If registered to Red Hat Satellite, the redhat.repo file points to Satellite's pulp instance, not the CDN. Satellite syncs content from CDN on its own schedule. A pool change on the portal does not immediately propagate to Satellite. The local system sees valid certificates but Satellite serves stale or missing content.

\n

Check: grep baseurl /etc/yum.repos.d/redhat.repo — if it contains satellite.example.com, verify Satellite's Content View promotion and Sync Plan status before troubleshooting the client.

\n\n

Conditions That Change the Design

\n

\n
• Air-gapped environments: No CDN access. Use Satellite or reposync to mirror content locally. Disable rhsmcertd (systemctl disable rhsmcertd) and manage repositories via static .repo files pointing to the internal mirror. The trust boundary shifts to the mirror's GPG signatures.
\n
• Image Builder / Golden Image pipelines: Do not register the build VM. Use subscription-manager register --activationkey only at runtime via cloud-init or ignition. The image contains no entitlement certificates; repositories are configured at first boot.
\n
• Simple Content Access (SCA) enabled accounts: Pool attachment is optional. subscription-manager attach --auto still works but repositories are granted based on the account's SCA entitlements, not specific pool IDs. Verification shifts to subscription-manager status --json | jq .subscription_status showing \"Simple Content Access\".
\n

\n\n

Concrete Example: Diagnosing a 403 on BaseOS

\n

Scenario: dnf install vim fails with 403 on rhel-9-for-x86_64-baseos-rpms.

\n

\n
1. subscription-manager status → Overall Status: Current
\n
2. subscription-manager list --consumed --pool-only → Shows pool 2c9180838a1234567890abcdef12345678
\n
3. subscription-manager repos --list-enabled --json | jq '.[] | select(.id==\"rhel-9-for-x86_64-baseos-rpms\")' → Returns empty (repo not enabled)
\n
4. cat /etc/yum.repos.d/redhat.repo | grep -A5 baseos → Shows enabled=0
\n
5. subscription-manager refresh → Repo enabled, dnf install vim succeeds
\n

\n

Root cause: Pool rebalance revoked BaseOS entitlement from the old pool. The 4-hour rhsmcertd cycle had not run. Manual refresh resolved it.

\n\n

Verification Checklist After Any Change

\n

\n
• subscription-manager status reports Overall Status: Current
\n
• subscription-manager repos --list-enabled output matches dnf repolist enabled line-for-line on repo IDs
\n
• dnf repoinfo shows SSL client cert: /etc/pki/entitlement/.pem and SSL client key: /etc/pki/entitlement/.pem
\n
• dnf makecache --refresh completes without 403/404 errors
\n

\n

If all four pass, the trust chain from portal → certificate → repository → DNF is intact.