Traefik Mesh CRD schema changes and controller downgrade compatibility

Direct Answer

Traefik Mesh does not deploy a conversion webhook for its CRDs as of v1.5 (the latest version before the 2024-06 knowledge cutoff). Downgrading the controller after a CRD schema change can leave existing custom resources in an unreadable or invalid state, depending on the nature of the schema change. Resources are not migrated in place automatically. Major version boundaries (e.g., v1.0 → v1.1, v1.2 → v1.3) should be treated as potentially non-reversible for CRD schema purposes.

Confirmed Behavior

• No conversion webhook: Traefik Mesh relies on Kubernetes' native spec.versions and storage version selection. The API server serves the stored version; it does not translate between versions on read.
• Schema additions (new optional fields): Downgrades often appear to work because the older controller ignores unknown fields. The resource remains readable, but new configuration is silently dropped.
• Schema removals or type changes: Downgrades frequently cause validation errors. The API server rejects resources containing fields the older schema does not recognize, or fields whose type no longer matches.
• Documentation guidance: Traefik Mesh release notes advise treating major/minor version bumps as potentially non-reversible for CRD schema and recommend avoiding downgrades across those boundaries unless the CRD manifests have been explicitly compared for backward compatibility.

Likely Explanation (Unverified)

The absence of a conversion webhook means Kubernetes stores each custom resource at a single storage version. When the controller upgrades, it may write resources using a newer version. On downgrade, the older controller can only serve/validate the versions it knows. If the stored version contains fields the older schema lacks, the API server returns a validation error on GET or LIST operations, effectively making the resource unreadable by that controller.

Steps to Verify for Your Versions

1. Export the CRD manifests for both the current and target controller versions:

   kubectl get crd meshservices.traefikmesh.io trafficsplits.traefikmesh.io meshpolicies.traefikmesh.io -o yaml > current-crds.yaml
   # Repeat after switching to the target version's manifests

2. Compare spec.versions, spec.versions[*].schema.openAPIV3Schema, and spec.versions[*].storage between the two manifests. Look for:
   • Removed fields
   • Changed field types (e.g., string → object)
   • Changed required lists
   • Storage version changes
3. Create a test resource using the newer version's fields, then downgrade the controller and attempt to read it:

   kubectl get meshservice test-service -o yaml
   # Observe: validation error, missing fields, or successful read

4. Check the release notes/changelog for both versions for keywords: "CRD conversion webhook", "storage version migration", "backward-compatible schema", "breaking change".

One Missing Diagnostic Detail

Which specific Traefik Mesh version pair are you considering for upgrade/downgrade? The exact field-level diff between those two versions determines whether a downgrade will succeed, fail with validation errors, or silently drop configuration. Without the version pair, I cannot confirm whether your specific crossing is reversible.