Question
x509: certificate signed by unknown authority in Teleport Proxy Service
Ash Atlas
0 reputation · 10 Jan 2021, 04:15 UTC
77.7K views
Loading…
0 reputation · 10 Jan 2021, 04:15 UTC
When deploying Teleport in a production environment, the Proxy Service must establish a trusted TLS connection to the Auth Service. In local tests the connection succeeds when the Proxy is started with --insecure or when the internal CA is manually added to the host trust store. In production, without --insecure and relying only on the system CA bundle, the Proxy logs the error x509: certificate signed by unknown authority. The unresolved decision is how to configure trust for the internal CA while avoiding insecure TLS bypasses and without altering the global system trust store.
Consider the trade‑offs between pointing the Proxy to the internal CA via --auth.ca-file versus re‑issuing the Auth Service certificate from a publicly trusted CA, taking into account certificate rotation, proxy restart requirements, and compliance with internal security policies.
Should the Proxy be configured with --auth.ca-file pointing to the internal CA bundle, or should the Auth Service be re‑issued a certificate from a public CA? What operational impact does each option have on certificate renewal, proxy restarts, and cluster‑wide trust management?
35275 reputation · 10 Jan 2021, 13:56 UTC
The Teleport Proxy must be able to validate the Auth Service’s TLS certificate. The most straightforward and policy‑conformant way to achieve this in production is to point the Proxy at the cluster’s internal CA bundle using the --auth.ca-file (or equivalent) flag. This keeps the Auth Service certificate signed by the internal CA, avoids external dependencies, and eliminates the need for a global system trust store change or the insecure flag.
| Aspect | Internal CA Bundle | Public CA Certificate |
|---|---|---|
| Certificate Rotation | Rotate the CA bundle; each Proxy must be updated and restarted (or reloaded if supported). | Auth Service certificate renews; Proxy may need a restart if it does not automatically reload the system CA store. |
| Proxy Restarts | Required after any CA bundle change. | Required after each Auth Service renewal. |
| Cluster‑wide Trust Management | Single CA bundle managed by configuration management; easy to audit. | Requires ensuring the system CA store on every host contains the public root. |
| Compliance | Matches internal PKI policies; no external root exposure. | May violate internal policy if public roots are disallowed for internal services. |
teleportteleport version and check teleport start --help for the exact flag name (e.g., --auth.ca-file or --ca-file in newer releases)./var/lib/teleport/ca.pem (or the path configured by ca_file in teleport.yaml) to a secure location accessible by all Proxy hosts./etc/teleport/ca.pem.teleport start --auth.ca-file=/etc/teleport/ca.pemopenssl s_client -connect auth.example.com:3025 -showcerts | openssl x509 -noout -issuer -subjectIf your organization’s policy mandates that all externally reachable services use public certificates, you may re‑issue the Auth Service certificate from a trusted public CA. In that case:
To fine‑tune the recommendation, please provide the Teleport release you are running (e.g., v12.1.0). The exact flag name and reload behavior vary between releases.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 10 Jan 2021, 09:42 UTC
One detail worth adding to the internal-CA recommendation: during rotation you do not have to swap the bundle atomically. The --auth.ca-file bundle can contain both the old and the new CA root, so Proxies keep trusting the Auth Service across the transition window. Since Go's x509 verification accepts a chain to any root in the bundle, a combined bundle lets you roll Proxies gradually instead of coordinating a cluster-wide restart.
Operationally, the flow is: export the CA with tctl auth export --type=tls, append the new root, distribute via config management, restart Proxies, then have the Auth Service switch to a cert signed by the new CA. Only after all Proxies have the combined bundle do you remove the old root from the bundle and roll again.
Also note the bundle must be readable by the teleport process — a permissions error on the CA file surfaces as the same x509: certificate signed by unknown authority, which can mislead you into debugging certificates when it is actually a file access problem. Check with sudo -u teleport head -c1 /path/to/ca-bundle.pem before touching the PKI.
Verify with openssl verify -CAfile bundle.pem auth-cert.pem that the Auth cert chains to a root in the bundle; this catches missing intermediates early — a frequent cause of the error is an intermediate cert present in the Auth Service's chain but absent from the Proxy's bundle.