Question
SSL certificate verification failed despite adding files to ca-certificatesFiles
Rin Dey
0 reputation · 19 Nov 2024, 01:04 UTC
125.3K views
Loading…
0 reputation · 19 Nov 2024, 01:04 UTC
In NixOS, the system-wide trust store is managed via the ca-certificates package and the security.ca-certificatesFiles option. While adding custom CA bundles to this configuration successfully updates the global store during a rebuild, certain runtimes and environments still fail to recognize these certificates during TLS handshakes.
The issue persists when language-specific environments, such as Java (JVM) or Python, look for hardcoded paths or standard locations like /etc/ssl/certs instead of the Nix store-managed paths generated by the system. Even when openssl s_client validates the chain using the system-wide store, these applications report 'certificate verify failed' errors.
How can one ensure that non-native runtimes consistently consume the Nix-managed CA store without manual environment variable overrides? Is there a native NixOS mechanism to point these environments to the certificates defined in security.ca-certificatesFiles?
34775 reputation · 19 Nov 2024, 08:05 UTC
The native NixOS mechanism is not a runtime configuration switch—it is a build-time dependency. Each language runtime (Java, Python, Node, etc.) must be built or wrapped against pkgs.cacert so its internal trust store is replaced with the Nix-managed bundle /etc/ssl/certs/ca-bundle.crt. security.ca-certificatesFiles only constructs that bundle; it does not rewrite language-specific stores.
nixos/etc/ssl/certs/ca-bundle.crt (a symlink into the Nix store) from the ca-certificates package plus every file listed in security.ca-certificatesFiles. OpenSSL, curl, wget, and any tool linked against the system OpenSSL pick this up automatically.cacerts keystore, Python bundles certifi, and Node carries its own root list. These are baked into their respective derivations at build time and are not re-read from /etc/ssl/certs at runtime unless explicitly patched.openssl s_client -CAfile /etc/ssl/certs/ca-bundle.crt), but the failing runtimes are using their embedded stores, which lack those CAs.pkgs.jre, pkgs.jdk, pkgs.openjdk). These derivations already merge the system cacert bundle into $JAVA_HOME/lib/security/cacerts at build time.pkgs.cacert to its runtime closure and launch with:
JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/etc/ssl/certs/ca-bundle.crt" java -jar app.jarpkgs.python3 (or pkgs.python3Packages.python). The nixpkgs derivation patches certifi so certifi.where() returns /etc/ssl/certs/ca-bundle.crt, and the ssl module's default context honors it.SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt in the environment or wrapper.pkgs.nodejs (or pkgs.nodejs-20_x, etc.). The derivation injects --use-openssl-ca at build time, making Node read the system bundle.NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle.crt.Enable the p11-kit trust module so GnuTLS-based tools (and any consumer of p11-kit) see the bundle automatically:
{ config, pkgs, ... }:
{
security.p11-kit.enable = true;
security.p11-kit.trustedPaths = [ "/etc/ssl/certs/ca-bundle.crt" ];
}This does not affect Java, Python, or Node directly.
For Docker containers built with dockerTools or systemd-nspawn services, bind-mount the host bundle:
# dockerTools.buildImage
contents = [ pkgs.cacert ]; # installs bundle inside image at /etc/ssl/certs/ca-bundle.crt
# or
runAsRoot = "ln -sf /etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt";openssl s_client -connect example.com:443 -CAfile /etc/ssl/certs/ca-bundle.crt — confirms system bundle validity.java -Djavax.net.ssl.trustStore=/etc/ssl/certs/ca-bundle.crt -version (or check cacerts file size matches system bundle).python3 -c "import ssl, certifi; print(ssl.get_default_verify_paths().cafile); print(certifi.where())" — both should print /etc/ssl/certs/ca-bundle.crt.cat /etc/ssl/certs/ca-bundle.crt | grep -c 'BEGIN CERTIFICATE' — count should equal Mozilla roots + your custom CAs.Which specific runtime(s) and version(s) are failing (e.g., openjdk-21 from nixpkgs, a downloaded jdk-17.tar.gz, python3.11 from python3Packages, a vendor Python)? The exact derivation determines whether the cacert patch is already applied or a wrapper is required.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 19 Nov 2024, 10:42 UTC
One distinction worth separating is the trust-store format itself. The NixOS bundle at /etc/ssl/certs/ca-certificates.crt is a PEM file that OpenSSL-based clients parse, whereas the JVM expects a binary keystore (JKS or PKCS12) and does not read PEM bundles through SSL_CERT_FILE. That format gap explains why a rebuild updating the system bundle can leave Java verification unchanged. Some Nixpkgs derivations generate a keystore and wrap the JVM to consume it, but this is package-specific and varies across Nixpkgs revisions, so verify against your pinned channel rather than assuming a universal option. For Python, whether the interpreter sees the Nix bundle depends on which OpenSSL it was linked against; a foreign or statically linked build may resolve to compiled-in paths instead. If a daemon still fails after a rebuild, restarting it can rule out cached directory entries or an in-memory keystore loaded at startup.