CentOS 8's systemd-resolved resolv.conf symlink: mixed resolver behavior and TLS trust mismatches

Short answer: on CentOS 8 there is no single "resolver" — there are at least three paths (glibc NSS, direct /etc/resolv.conf readers, and non-glibc resolvers), plus a separate trust-store split between the p11-kit system bundle and the JVM's own cacerts. You diagnose contradictions by testing each path independently, not by assuming the symlink unifies behavior.

Note on scope: this describes final CentOS 8 (now EOL) behavior with its shipped systemd/NetworkManager versions. CentOS Stream and RHEL rebuilds may differ; verify against your installed packages.

1. Determining which resolver path an application uses

Confirmed facts first: /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, pointing at the 127.0.0.53 stub listener. But reading that file and using systemd-resolved are different things:

• glibc apps follow the hosts: line in /etc/nsswitch.conf. With resolve listed, they talk to systemd-resolved over its varlink/D-Bus interface directly; with dns, they read resolv.conf and hit the stub.
• dig/nslookup/host bypass NSS entirely and always query the server in resolv.conf (127.0.0.53) unless told otherwise.
• Statically linked or pure-Go binaries may ignore nsswitch.conf completely and parse resolv.conf themselves.

The practical test is a three-way comparison for the problem hostname:

getent hosts app.example.com          # NSS path (what most glibc apps see)
dig @127.0.0.53 app.example.com       # stub listener path
dig @192.0.2.10 app.example.com       # your actual upstream DNS

If getent and the stub disagree, NSS is configured differently than you think — check nsswitch.conf. If the stub and upstream disagree, systemd-resolved is using a different link's DNS server than expected; run resolvectl status to see per-link DNS servers. If all three agree but one specific binary still fails, that binary is bypassing NSS (common with Go's pure resolver) — check its docs or force the cgo resolver.

2. Stale cache after a NetworkManager-triggered restart

Normally a full restart of systemd-resolved drops its cache, so persistent stale answers after a restart are more likely a per-link server change than cache. Before blaming upstream DNS, confirm with counters rather than guesswork:

resolvectl statistics          # note cache hit/miss counts
resolvectl flush-caches
getent hosts app.example.com   # re-test
resolvectl statistics          # misses should have incremented

If flushing changes the answer, the cache was the culprit. If it doesn't, compare the stub query against the upstream query from step 1 — that isolates whether systemd-resolved is forwarding to the wrong server (a NetworkManager/link configuration issue) rather than serving stale data. Do not edit /etc/resolv.conf directly; as a managed symlink, changes are overwritten and can break the stub setup.

3. TLS failing only in Java: which trust bundle?

A hostname that resolves fine but fails TLS verification only in Java almost always means the JVM is validating against a different anchor set than the system. CentOS 8's system trust is maintained by update-ca-trust/p11-kit, with extracted bundles under /etc/pki/ca-trust/extracted/ — including a Java keystore at extracted/java/cacerts. Whether your JVM actually uses it depends on vendor packaging: some distro OpenJDK builds symlink the JVM cacerts to the p11-kit bundle; others ship their own file at $JAVA_HOME/lib/security/cacerts (or jssecacerts). Don't assume — check:

ls -l $JAVA_HOME/lib/security/cacerts        # symlink to /etc/pki/... or a real file?
trust list | grep -i "YourCA"                 # is the CA in the system store?
keytool -list -cacerts -storepass changeit | grep -i "YourCA"

Cross-check behavior: if curl -v https://app.example.com succeeds (system store) while the Java app fails, the trust stores diverge. Then either add the CA via update-ca-trust (place the PEM in /etc/pki/ca-trust/source/anchors/ and run update-ca-trust extract) if the JVM is linked to the system bundle, or import it into the JVM keystore — or point the app at the extracted bundle with -Djavax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts.

One detail that changes the recommendation

If the failing Java process runs in a container, none of the host-level checks apply — the container image carries its own resolv.conf, nsswitch.conf, and cacerts. Confirm whether the app is containerized before modifying host trust.