Which conditions trigger Firefox to fall back to system DNS after a DoH certificate error?

Short answer

A certificate validation error on the DoH endpoint does not trigger a silent fallback to the system resolver in current Firefox builds. Fallback is gated by network.trr.mode, and certificate failures are classified in the Trusted Recursive Resolver (TRR) code path as security errors, not as the transient network or server errors that fallback exists to absorb. This is the likely behavior based on how the mode policy and error classification are structured; it is version-sensitive, so verify against your specific build before relying on it.

What is confirmed vs. likely

Confirmed: DoH in Firefox runs through the TRR subsystem, and network.trr.mode controls the fallback policy:

• Mode 1 (DoH first): DoH is preferred, and Firefox may fall back to the system resolver on failure.
• Mode 2 (DoH only): no fallback is permitted; a DoH failure means resolution fails from the application's perspective.

Likely, pending verification on your build: in mode 1, the fallback path is designed for transient conditions — timeouts, unreachable servers, malformed responses. A TLS certificate validation failure is treated as a non-recoverable security failure, so the query is failed rather than downgraded to plain system DNS. In mode 2 the outcome is the same (no fallback) but for a different reason: fallback is disabled entirely.

Answering the unresolved questions

1. Security failure vs. transparent fallback

The deciding factor is error classification, not the mode alone. Mode 1 permits fallback, but only for errors the TRR layer considers transient. A certificate error indicates the endpoint's identity could not be verified; silently downgrading to unencrypted system DNS after that would defeat the purpose of DoH, so it is handled as a hard failure. Treat any claim that "all TLS errors fall back" as outdated — this classification has shifted across versions.

2. Caching and logging of the bad certificate

Firefox does not surface a certificate-override dialog for DoH endpoints the way it does for websites, and there is no user-facing cache of the invalid DoH certificate to inspect. Diagnostics are exposed indirectly through TRR counters and console logging rather than a certificate store. The practical place to look is the internal networking/DNS diagnostics page and the Browser Console with TRR-related logging enabled.

3. Channel and OS differences

Nightly, Beta, and Release are believed to share the same mode-gated fallback policy; differences are in logging verbosity and default preferences, not in the core decision. The OS affects how the system resolver is reached once fallback is permitted, but not whether a certificate error permits it. Because defaults and classification have changed across releases, reproducibility requires pinning both the build and the network.trr.* preferences.

How to verify on your build

1. Open about:config and record network.trr.mode and network.trr.uri.
2. Point the URI at a test DoH endpoint presenting an invalid or self-signed certificate.
3. Watch the Browser Console for TRR error messages and check the DNS diagnostics counters for success/failure after lookups.
4. Repeat with mode 1 and mode 2. If resolution succeeds via the system resolver in mode 1 after the certificate error, your build's classification differs from the expected behavior — report the exact version.

One caveat: because this behavior is version-sensitive, the single most useful diagnostic detail is the exact Firefox version and channel where you observed a fallback after a certificate error — that determines whether you're seeing a regression or intended older behavior.