GTlsClientConnection rejects valid certificates in Vala: diagnosing validation and DNS lookup behavior

Direct Answers

1. Retrieving rejection flags: When validate-certificates is true and the handshake fails, connect to the accept-certificate signal. In the handler, call certificate.verify() on the supplied GTlsCertificate. The returned GTlsCertificateFlags value contains the exact reason — e.g., G_TLS_CERTIFICATE_UNKNOWN_CA, G_TLS_CERTIFICATE_EXPIRED, G_TLS_CERTIFICATE_BAD_IDENTITY. Return false from the handler to abort the connection; returning true would override the failure, which defeats the purpose of validation. Note that if the connection is aborted before the signal fires (for example, a network timeout), you must instead check the async operation result for a TLS handshake error.
2. CA bundle handling: Yes, call set_ca_file() (or set_ca_directory()) before the handshake if the system bundle is missing or outdated. This supplies the trust anchors. It works independently of validate-certificates, but validation only occurs when that property is true, so both must be configured together for proper chain checking. After setting, you can confirm the active bundle with get_ca_file(), since on some platforms the default path is fixed at compile time.
3. DNS-before-TLS ordering: Gio.Resolver.lookup_by_name_async completes its callback before the TLS handshake begins on the resulting connection. The resolver returns an IP address, and only then does the connect and handshake proceed. In practice this ordering holds because the handshake cannot start without a connected socket, which requires a resolved address.

Recommended Connection Sequence

var resolver = Resolver.get_default();
resolver.lookup_by_name_async(hostname, null, (obj, res) => {
    try {
        var addresses = resolver.lookup_by_name_finish(res);
        var client = new SocketClient();
        // connect using the resolved addresses, then wrap in TLS
        var tls = new TlsClientConnection(conn);
        tls.validate_certificates = true;   // MUST be set before handshake
        tls.set_ca_file("/etc/ssl/certs/ca-certificates.crt"); // if needed
        tls.accept_certificate.connect((tls_conn, cert) => {
            var flags = cert.verify();
            if (flags != 0) {
                stderr.printf("TLS verification failed: %u\n", (uint) flags);
                return false; // reject
            }
            return true;
        });
    } catch (Error e) {
        // DNS error: G_IO_ERROR_NOT_FOUND or G_IO_ERROR_TIMED_OUT
    }
});

Distinguishing DNS vs TLS Failures

• DNS failure: The lookup_by_name_async callback receives an error — typically G_IO_ERROR_NOT_FOUND (NXDOMAIN) or G_IO_ERROR_TIMED_OUT. No TCP connection is attempted and no TLS signal fires.
• TCP/TLS failure: The connect or handshake callback receives an error. A verification failure surfaces as a TLS handshake error from the finish call, and the accept-certificate signal will have fired with the problematic certificate.
• Debug logging: G_DEBUG=resolver and G_DEBUG=tls are useful for observability but do not affect programmatic flow. Rely on callback errors and signals for logic.

Verification Checklist

1. Test against a known-good host (e.g., www.gnutls.org:443) with validate_certificates = true — expect zero flags from cert.verify().
2. Test against a self-signed or expired certificate (local test server) — confirm G_TLS_CERTIFICATE_UNKNOWN_CA or G_TLS_CERTIFICATE_EXPIRED appears.
3. Disable networking or resolve a non-existent name — confirm the DNS error arrives before any TLS signal.

One Diagnostic Detail Needed

Which GLib/GIO version are you targeting? The accept-certificate signal and GTlsCertificate.verify() behavior described here assume a reasonably recent GLib; older releases differ in the verification API. Knowing your minimum version lets me confirm the exact method names and flag constants.