Choosing Between Keg‑Only Isolation and Force‑Linking for Homebrew OpenSSL, Python, and LLVM on macOS 13+ and Linuxbrew

Decision and constraints

Homebrew marks several core formulae — openssl@3, python@3.12, llvm, libxml2 — as keg‑only. They are installed under /opt/homebrew/Cellar/<name>/<version> (Apple Silicon) or /usr/local/Cellar/… (Intel) but are not symlinked into the main prefix (/opt/homebrew or /usr/local). This prevents shadowing system libraries and avoids version conflicts between Homebrew packages.

The decision you face when a consumer (your own project, a third‑party build, or a CI job) needs headers or libraries from a keg‑only formula is:

• Accept the default isolation and pass explicit paths at build time.
• Force‑link the formula into the prefix with brew link --force.
• Use Homebrew’s built‑in pkg‑config integration (since 3.6.0) to inject paths automatically.

Constraints include macOS 13+ (Ventura) system LibreSSL incompatibility with OpenSSL 3, Apple Silicon vs. Intel prefix differences, and Linuxbrew’s identical keg‑only behavior but fewer system‑library clashes.

Supported options compared

Approach	How it works	Pros	Cons / Risks
Default keg‑only (explicit flags)	Pass -I$(brew --prefix openssl@3)/include -L$(brew --prefix openssl@3)/lib or set PKG_CONFIG_PATH in the build script.	Reproducible, no global side‑effects; works on macOS and Linuxbrew; supported by Homebrew maintainers.	Requires changes to every build configuration (CMake, Make, Meson, etc.).
brew link --force <formula>	Creates symlinks in /opt/homebrew/{bin,lib,include} so the formula becomes globally discoverable.	Zero changes to consumer build files; simple for ad‑hoc development.	Unsupported — issues closed as user‑error. Can break other Homebrew packages (e.g., python@3.11 linked to OpenSSL 1.1). On macOS, may cause segfaults in system tools that dlopen Homebrew libraries. Overwrites python3 symlink managed by another Python version.
brew link --overwrite <formula> (selective)	Only overwrites conflicting files instead of the whole keg.	Reduces scope of conflict compared to --force.	Still unsupported; partial linking can leave inconsistent state.
Automatic pkg‑config integration	Homebrew’s build environment adds keg‑only pkgconfig directories to PKG_CONFIG_PATH during brew install of a consumer formula.	No manual flags for Homebrew‑managed consumers; works for brew install wget, curl, etc.	Only active inside Homebrew’s sandbox; not available for arbitrary external builds unless you replicate the environment.
Explicit environment variables in CI / custom builds	Export PKG_CONFIG_PATH="$(brew --prefix openssl@3)/lib/pkgconfig:$(brew --prefix zlib)/lib/pkgconfig:$PKG_CONFIG_PATH" and use $(brew --prefix openssl@3) in CMake/Make configure flags.	Fully reproducible, works outside Homebrew sandbox, no force‑linking.	Must be maintained in each CI pipeline or developer machine setup.

Trade‑off analysis

Stability vs. convenience

Force‑linking gives immediate convenience but introduces fragile global state. Homebrew maintainers explicitly discourage it; any breakage is considered user error. The explicit‑flags and environment‑variable approaches keep the prefix clean and make dependencies visible in build logs, which is essential for reproducible CI/CD.

System library interactions

macOS ships LibreSSL, which is not ABI‑compatible with Homebrew’s OpenSSL 3. Force‑linking OpenSSL 3 into the prefix can cause system utilities that load Homebrew‑linked libraries (via dlopen) to crash. On Linuxbrew, the system package manager (apt, dnf) often provides compatible OpenSSL, so conflicts are rarer but still possible when mixing Homebrew and system packages.

Python version management

python@3.12 is keg‑only because it only installs a python3.12 binary. Force‑linking would overwrite the python3 symlink that may point to python@3.11 or the system Python, breaking scripts that rely on a specific version.

Compiler toolchain (LLVM)

llvm is keg‑only to avoid clashing with Xcode’s clang. Force‑linking can make clang resolve to Homebrew’s LLVM, which may have different default target triples or sanitizer runtimes, leading to subtle build failures.

Concrete implementation: reproducible build without force‑linking

The following snippet works on both macOS (Apple Silicon or Intel) and Linuxbrew. Place it in your CI script (GitHub Actions, GitLab CI, Azure Pipelines) or a local build.sh.

#!/usr/bin/env bash
set -euo pipefail

# 1. Resolve Homebrew prefix (works on both architectures)
BREW_PREFIX="$(brew --prefix)"

# 2. Export pkg‑config paths for keg‑only dependencies
OPENSSL_PREFIX="$(brew --prefix openssl@3)"
ZLIB_PREFIX="$(brew --prefix zlib)"
export PKG_CONFIG_PATH="${OPENSSL_PREFIX}/lib/pkgconfig:${ZLIB_PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH:-}"

# 3. Example CMake configure step
cmake -S . -B build \
  -DOPENSSL_ROOT_DIR="${OPENSSL_PREFIX}" \
  -DZLIB_ROOT="${ZLIB_PREFIX}" \
  -DCMAKE_PREFIX_PATH="${BREW_PREFIX}"

# 4. Build
cmake --build build --config Release

Required permissions: a regular user account with Homebrew installed; no sudo needed. The script assumes Homebrew 4.2+ (provides brew link --dry-run and brew info --json=v2) and macOS 13+ or a glibc 2.35+ Linux distribution.

Validation steps

1. Confirm keg‑only status

   brew info openssl@3 | grep "keg-only"
   # Expected output: "keg-only: true"
   

2. Verify prefix contents

   ls -l "$(brew --prefix openssl@3)/include/openssl/opensslv.h"
   ls -l "$(brew --prefix openssl@3)/lib/libssl.3.dylib"   # macOS
   # or
   ls -l "$(brew --prefix openssl@3)/lib/libssl.so.3"      # Linux
   

3. Test pkg‑config integration

   PKG_CONFIG_PATH="$(brew --prefix openssl@3)/lib/pkgconfig" \
     pkg-config --cflags --libs openssl
   # Should print something like:
   # -I/opt/homebrew/opt/openssl@3/include -L/opt/homebrew/opt/openssl@3/lib -lssl -lcrypto
   

4. Compile a minimal test program

   echo '#include ' | \
     clang -x c - -o /dev/null \
     -I"$(brew --prefix openssl@3)/include" \
     -L"$(brew --prefix openssl@3)/lib" -lssl -lcrypto
   # Exit code 0 indicates success.
   

5. Check runtime linkage of a Homebrew consumer

   brew install curl
   # macOS
   otool -L "$(brew --prefix curl)/bin/curl" | grep -E 'ssl|crypto'
   # Linux
   ldd "$(brew --prefix curl)/bin/curl" | grep -E 'ssl|crypto'
   # Output should show the Homebrew OpenSSL paths, not system LibreSSL.
   

If any step fails, revisit the exported paths or ensure the formula is installed (brew install openssl@3 zlib).

Limitations and practical checks

• Force‑linking is not a supported configuration; any subsequent brew doctor will warn about it.
• The explicit‑flags method requires every consumer build system to accept custom include/library paths. Some legacy Makefiles may need patching.
• On macOS, the system Python (in /usr/bin/python3) remains untouched; only Homebrew‑managed Python versions are affected by keg‑only policy.
• Linuxbrew installs to ~/.linuxbrew or /home/linuxbrew/.linuxbrew; replace /opt/homebrew with $(brew --prefix) in scripts to stay portable.

Practical verification: after a successful build, run the linkage check (step 5 above) on the produced binary. If it resolves to the Homebrew OpenSSL paths, the isolation strategy works; if it shows system LibreSSL, the build picked up the wrong library and you must adjust PKG_CONFIG_PATH or CMake flags.

Diagram labels

• Keg‑only install
• Force‑link symlinks
• Build env vars
• Runtime linkage check