Question
k3os CoreDNS upstream resolution and CA trust store propagation
Cig Finch
0 reputation · 12 Nov 2021, 07:11 UTC
views
Loading…
0 reputation · 12 Nov 2021, 07:11 UTC
In a k3os environment, CoreDNS operates as a system-wide addon managed by the kube-controller. When deploying on edge nodes with specific upstream DNS providers, resolution often fails because the node's resolv.conf conflicts with the CoreDNS configuration in the k3s manifest.
Furthermore, because k3os utilizes a read-only root filesystem by default, injecting custom CA certificates into the host trust store to facilitate TLS validation is non-trivial. This results in certificate validation failures for containerized applications that rely on the built-in CA generated during initial boot, as the trust anchor is not propagated to downstream pods via standard file system paths.
What is the recommended method to persistently configure upstream DNS forwarders in the k3s manifest without them being overwritten by the agent on reboot? How can custom CA certificates be injected into the k3os trust store while respecting the read-only nature of the OS?
35275 reputation · 12 Nov 2021, 15:58 UTC
To persistently configure upstream DNS forwarders in k3os without them being overwritten by the node's resolv.conf, you must modify the CoreDNS ConfigMap to use explicit IP addresses instead of the default /etc/resolv.conf reference. To handle CA certificates on a read-only filesystem, you must utilize a HostPath mount or a ConfigMap to inject certificates into the pods, as the read-only root prevents direct modification of the host's trust store.
By default, CoreDNS in k3s/k3os uses the forward . /etc/resolv.conf directive. Because the k3s agent may synchronize the node's resolver settings, the most stable method is to hardcode the upstream providers in the CoreDNS configuration.
k3oskubectl get configmap coredns -n kube-system -o yaml > coredns.yamlforward . /etc/resolv.conf line in the Corefile./etc/resolv.conf with your specific upstream DNS IPs (e.g., forward . 8.8.8.8 1.1.1.1).kubectl apply -f coredns.yamlNote: To ensure these changes survive cluster-wide updates or k3os re-imaging, these modifications should be managed via a GitOps pipeline or a k3s manifest file located in /var/lib/rancher/k3s/server/manifests/.
Since the k3os root filesystem is read-only, attempting to write to /etc/ssl/certs on the host will fail. The recommended approach is to propagate the trust anchor at the pod level.
/etc/ssl/certs/ca-certificates.crt (or the distribution-specific path)./var), mount that specific directory into the pod using a hostPath volume.Verify DNS resolution from within a pod using a scoped command:
kubectl exec -it <pod-name> -- nslookup google.comVerify CA propagation by checking the certificate bundle inside the container:
kubectl exec -it <pod-name> -- ls /etc/ssl/certs/To provide a more specific mount path for the CA certificates, please specify if your custom CA is stored in a persistent /var subdirectory or if it is provided as an external file during the k3os deployment process.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 12 Nov 2021, 17:34 UTC
CoreDNS upstream behavior in k3s is driven by the Corefile ConfigMap in kube-system, not directly by the node's /etc/resolv.conf. The forwarder plugin defines which nameservers CoreDNS uses for external queries; if no explicit forwarders are set, CoreDNS falls back to the nameservers in its own pod resolv.conf, which mirrors the node.
For k3os read-only root, persistent host trust store changes are not durable across reboots. CA trust for workloads is typically handled per-pod via a ConfigMap mounted into /etc/ssl/certs or via an init step that updates the container's trust store, rather than modifying the host store.