Implementing Canary Traffic Splitting with Traefik Mesh

Solving the Zero-Downtime Deployment Risk

Updating a production service by replacing all pods at once creates a high-risk window where a single bug can cause a total outage. The goal is to shift a small percentage of traffic to a new version (the canary) to validate stability before a full rollout. In Traefik Mesh, this is achieved by decoupling the Kubernetes Service from the actual traffic distribution, using a sidecar proxy to handle weighted routing at the data plane level.

How Traefik Mesh Intercepts Traffic

Unlike standard Kubernetes Services that use kube-proxy for simple round-robin load balancing, Traefik Mesh injects a sidecar proxy into every pod. When Service A calls Service B, the request is intercepted by Service A's sidecar. This proxy consults the mesh control plane to determine which version of Service B should receive the request based on defined weights, rather than relying on the cluster's DNS or IP tables.

Configuration Example: 90/10 Traffic Split

To implement a canary release, you must define the different versions of your service as separate Kubernetes Deployments but keep them under a unified mesh identity. The following example demonstrates a traffic split where 90% of requests go to the stable version (v1) and 10% go to the canary (v2).

# Apply this Mesh routing configuration via kubectl
# Required Permissions: Cluster-admin or permissions to edit Mesh CRDs

apiVersion: mesh.traefik.io/v1alpha1
kind: TrafficSplit
metadata:
  name: product-service-canary
spec:
  service: product-service
  backends:
    - name: product-service-v1
      weight: 90
    - name: product-service-v2
      weight: 10

Execution Details: Run this command on a workstation with kubectl configured for a cluster running Traefik Mesh. The service field refers to the virtual service name used by other pods, while backends refers to the actual Kubernetes services backing the specific versions.

Verifying the Traffic Distribution

Because the routing happens inside the sidecar, standard Kubernetes service logs will not show the split. You must verify the result through the data plane or application logs.

• Log Analysis: Check the logs of the v1 and v2 pods. For every 100 requests, v2 should show approximately 10 entries.
• Request Testing: Use a loop to send requests and count the responses: for i in {1..100}; do curl -s http://product-service | grep "version"; done | sort | uniq -c
• Proxy Inspection: Inspect the sidecar proxy logs for the calling service to confirm the destination IP matches the v2 pod.

Operational Limitations and Risks

While traffic splitting provides safety, it introduces specific engineering trade-offs:

• Resource Overhead: Every pod requires a sidecar proxy. In clusters with hundreds of microservices, this significantly increases the total CPU and RAM consumption across the namespace.
• Network Latency: Each request now travels through two proxies (outbound from source, inbound to destination), adding a small but measurable amount of latency to every internal call.
• Debugging Complexity: Standard tools like tcpdump on the node may show traffic moving between pods, but the logic governing why a request went to v2 instead of v1 is hidden within the mesh control plane and proxy configuration.

Common Implementation Mistakes

Overlapping Selectors: A common error is having the main Kubernetes Service selector match both v1 and v2 pods while simultaneously using a TrafficSplit. This can lead to "double balancing," where kube-proxy and Traefik Mesh both attempt to distribute traffic, resulting in unpredictable ratios.

Ignoring Readiness Probes: If the canary (v2) is not fully ready but the TrafficSplit is applied, 10% of your users will receive 503 errors. Always ensure the canary deployment is Ready before updating the weight.

Rollback Procedure

If the canary version exhibits errors, immediately revert the traffic to the stable version by updating the weights in the TrafficSplit resource:

# Set v1 to 100% and v2 to 0%
kubectl patch trafficsplit product-service-canary --type='merge' -p '{"spec":{"backends":[{"name":"product-service-v1","weight":100},{"name":"product-service-v2","weight":0}]}}'