Using Traefik Mesh TrafficSplit for Safe Canary Deployments in Kubernetes

Problem: shifting traffic without touching code

When you roll out a new version of a service you often want to send only a fraction of live traffic to it, watch metrics, and then increase the share if everything looks healthy. Doing this with a traditional load balancer usually requires either code changes (feature flags) or external configuration that lives outside the cluster.

Thesis

Traefik Mesh’s TrafficSplit custom resource lets you define weighted routing between multiple backends of a service directly inside the mesh. The Envoy sidecars that Traefik Mesh injects enforce the split in real time, so you can experiment with canary releases without modifying applications or external load balancers.

How TrafficSplit works

The TrafficSplit CRD references a Kubernetes service and lists one or more backends. Each backend points to a subset of pods (usually selected by labels) and carries a weight. The weights must total 100; the mesh interprets them as percentages of requests that should be sent to each subset. When you update the object, the mesh controller pushes the new weights to the Envoy proxies, which immediately rebalance traffic—no pod restarts are needed.

Prerequisites

• Traefik Mesh installed in the cluster with sidecar injection enabled.
• The target service must be registered with the mesh (i.e., its pods have the mesh sidecar). Services outside the mesh are ignored by TrafficSplit.
• You need kubectl access with permission to create and update custom resources in the namespace where the service lives (typically edit or higher).

Worked example: canary shift for an NGINX service

Assume you have two NGINX deployments representing versions v1 and v2. Both expose port 80 and are labeled app=demo with version=v1 or version=v2. They are backed by a single Kubernetes service called demo-svc in namespace example.

1. Label the deployments (if not already done):

   kubectl label deployment demo-v1 -n example version=v1
   kubectl label deployment demo-v2 -n example version=v2
   

2. Create or verify the service selects both versions:

   kubectl get svc demo-svc -n example -o yaml
   # Ensure selector includes app=demo
   

3. Apply a TrafficSplit that sends 80 % to v1 and 20 % to v2:

   cat <<EOF | kubectl apply -f -
   apiVersion: traefikmesh.io/v1alpha1
   kind: TrafficSplit
   metadata:
     name: demo-split
     namespace: example
   spec:
     service: demo-svc
     backends:
     - service: demo-svc
       weight: 80
       tags:
         version: v1
     - service: demo-svc
       weight: 20
       tags:
         version: v2
   EOF
   

4. Check that the object is accepted:

   kubectl get trafficsplit demo-split -n example -o yaml
   # Look for status: "Accepted" or similar
   

5. Generate traffic (run from a pod or your workstation with cluster access):

   for i in {1..100}; do curl -s http://demo-svc.example.svc.cluster.local/ > /dev/null; done
   

6. Verify the split via Prometheus (assuming the mesh exposes traefikmesh_requests_total):

   # Query for requests per version over the last minute
   sum by (version) (increase(traefikmesh_requests_total{service="demo-svc"}[1m]))
   

   You should see approximately 80 % of the increments labeled version=v1 and 20 % labeled version=v2. Small deviations are normal due to request granularity.

Trade‑offs and limitations

• Mesh‑only scope: If any part of your traffic bypasses the sidecar (e.g., an external load balancer that talks directly to the service pods), the split will not apply to that traffic.
• Weight sum must be 100: A misconfigured split (e.g., weights summing to 90) results in the missing 10 % being dropped, which can look like unexplained errors.
• No built‑in request‑level filtering: The split works on the service level; you cannot split based on headers or URLs without additional middleware.
• Operational overhead: You must keep the sidecar injection in sync with deployments; otherwise new pods may not receive the proxy and will fall outside the split.

Practical way to catch misconfigurations early: run kubectl apply --dry-run=client -f trafficplit.yaml before applying, and periodically audit that the sum of weights equals 100 with a simple script or jq query.

Actionable closing

Start by labeling your existing deployments, create a service that selects all versions, and apply a modest TrafficSplit (e.g., 90/10). Monitor the Prometheus metric traefikmesh_requests_total to confirm the distribution matches your intent. If the numbers look correct, gradually increase the weight of the new version. Should you need to revert, simply edit the TrafficSplit object to restore the original weights or delete it—the mesh will re‑program the sidecars instantly, rolling traffic back without any pod restarts.