Limits: Custom DNS resolver vs Host header mismatch in Ktor client

Short answer

No — there is no documented Ktor-level API to inject a custom DNS resolver into the Jetty engine, and a Host header override can never fix this. The clean solution is to keep the domain in the request URL and switch to the OkHttp engine, whose Dns interface is a documented extension point that maps the domain to your IP while TLS SNI and hostname verification continue to use the domain.

Why the Host header approach fails

TLS hostname verification happens during the handshake, against the hostname used to open the connection — i.e., the host in the request URL. The Host header is application-layer data sent after the handshake completes, so it plays no role in certificate validation. If your URL contains a raw IP, the certificate must contain that IP as a SAN entry; otherwise verification fails no matter what header you set. This is confirmed TLS behavior, not a Ktor quirk.

The likely explanation for what you observed: with the Jetty engine, Ktor delegates resolution to the JVM's InetAddress resolver, and there is no public hook to replace it per-client. You would have to reach into engine-internal socket factory wiring, which is version-sensitive across Ktor 2.x/3.x and fragile.

Recommended pattern: OkHttp engine with a Dns override

OkHttp separates which hostname is verified (always the URL host) from which address is dialed (the Dns lookup). That is exactly your requirement:

val client = HttpClient(OkHttp) {
    engine {
        config {
            dns(object : Dns {
                override fun lookup(hostname: String): List<InetAddress> =
                    if (hostname == "api.example.com")
                        listOf(InetAddress.getByName("203.0.113.10"))
                    else Dns.SYSTEM.lookup(hostname)
            })
        }
    }
}
// Request the domain, not the IP:
client.get("https://api.example.com/health")

SNI carries api.example.com, the certificate is verified against that name, and the connection goes to your chosen IP. Certificate validation stays fully enabled.

Testing without weakening validation

• Generate a local CA and a server certificate whose SAN contains only the test domain (not the IP). Add the CA to the client's trust store explicitly — never set validateCertificates = false, since that masks the exact failure mode you are testing.
• Map the domain to 127.0.0.1 in your Dns override and assert the request succeeds. Then re-run with a certificate missing the domain SAN: it must fail. That negative test proves hostname checks are still active.
• Confirm SNI carries the domain by enabling -Djavax.net.debug=ssl:handshake or capturing the handshake, and log the address your resolver returns versus the address actually dialed.

Caveats

Pinning a production domain to a fixed IP bypasses DNS-based failover and load balancing — treat it as an operational decision with a maintenance cost. Also verify the exact Dns wiring against your OkHttp/Ktor version; engine config lambdas differ between Ktor 2.x and 3.x. If you are pinned to the Jetty engine, the fallback is a custom SSLSocketFactory that dials the fixed IP while SSL parameters retain the domain, but that is more code and more risk than switching engines.