Traefik Mesh controller ↔ Traefik Proxy: config parse safety after controller downgrade
0 reputation · 21 Jul 2021, 04:29 UTC
0 reputation · 21 Jul 2021, 04:29 UTC
When a Traefik Mesh controller is upgraded to a version that introduces a new CRD schema (for example, a revised TrafficSplit definition), the controller continues to serve both the old and new versions while the Traefik Proxy only understands the schema version it was compiled against.
If the controller is later rolled back to the pre‑upgrade version, any Custom Resources that were written using the new schema may remain in the cluster in a format the proxy cannot parse, potentially causing config reload failures.
The goal is to determine what guarantees or mechanisms are needed to keep the proxy operational after such a downgrade without manual intervention.
34775 reputation · 21 Jul 2021, 10:51 UTC
No, a downgrade is not automatically safe, and no, you cannot rely on the proxy ignoring unknown fields. Kubernetes performs no automatic schema conversion on downgrade unless a conversion webhook exists — and Traefik Mesh does not ship one for its CRDs (TrafficSplit, TrafficTarget, HTTPRouteGroup, TCPRoute). Without a webhook, the API server does only a trivial apiVersion rename, so fields that exist only in the new schema are silently dropped when the old controller reads the object. The safe path is to convert or recreate affected CRs in the old schema before rolling back the controller.
One clarification to the premise: Traefik Proxy (the data plane) does not parse the raw CRDs. It consumes dynamic configuration derived and pushed by the Mesh controller. The parse-failure risk lives in the controller's informers and listers decoding persisted CRs into typed structs compiled against the old schema. That distinction matters operationally: if the old controller cannot decode a stored object, it may push an empty or partial configuration to the proxies — which can drop routes cluster-wide rather than merely log an error. So the blast radius is larger than "one unparseable CR."
Unknown-field tolerance only covers the easy case — additive changes. It fails when:
Note also that preserveUnknownFields was removed in Kubernetes 1.22+, so pruning behavior is governed by the structural schema — fields not in the served old schema are dropped on write, which is silent data loss, not a parse error you can catch.
There is no provider-mechanism skew tolerance to lean on here. Traefik Mesh has been deprecated by its maintainers and reached end-of-life, so no upstream rollback tooling or compatibility matrix should be expected. Any skew policy must be enforced by your own deployment pipeline.
kubectl get crd trafficsplits.split.smi-spec.io \
-o jsonpath='{.status.storedVersions}' If the new version appears, plan a storage migration before rollback.kubectl get crd trafficsplits.split.smi-spec.io \
-o jsonpath='{.spec.conversion}'cannot unmarshal or unknown field decode errors from informers.Exact CRD versioning varies by Mesh release — verify against the manifests shipped with the specific versions involved rather than assuming multi-version serving. Given the project's EOL status, treat this as a migration argument as much as a rollback question.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.