Canary Deployments in Traefik Mesh: HTTPRoute Traffic Splitting Without Sidecars

Concrete Problem: Lightweight Canary Releases

When a team rolls out a new version of a microservice, the classic approach is to deploy a separate pod, expose it via a Service, and redirect a fraction of traffic to it. In a sidecar‑based mesh like Istio, this requires adding a proxy container to every pod, which can add latency, memory overhead, and operational complexity. For teams that need rapid, low‑footprint canaries, a sidecar‑less solution that still offers fine‑grained traffic control is desirable.

Traefik Mesh Architecture in a Nutshell

Traefik Mesh eliminates the per‑pod sidecar by running a single Traefik Proxy instance as a DaemonSet on every node. All pods on that node route their traffic through the node‑local proxy. The Mesh Controller watches Kubernetes resources and programs these proxies, handling service discovery, mTLS, and traffic policies. This architecture reduces resource usage and simplifies deployment, but it also introduces a shared data plane that can become a single point of failure.

HTTPRoute‑Based Traffic Splitting for Canary Deployments

The core feature that lets Traefik Mesh perform canary releases is the HTTPRoute Custom Resource Definition (CRD). An HTTPRoute can reference multiple backend services and assign each a weight. The Mesh Controller translates these weights into routing rules inside the node‑local proxy. For example, a 90/10 split between a stable and a canary deployment is expressed as:

apiVersion: traefik.io/v1alpha1
kind: HTTPRoute
metadata:
  name: myapp-route
spec:
  routes:
    - match: Host("myapp.example.com")
      services:
        - name: myapp-stable
          port: 80
          weight: 90
        - name: myapp-canary
          port: 80
          weight: 10

When the controller applies this configuration, the proxy will forward 90% of incoming requests to myapp-stable and 10% to myapp-canary. The weights are rounded to the nearest integer, so the split is approximate but stable over time.

Key Points to Remember

• Traffic splitting only applies to HTTP/HTTPS. For TCP/UDP you need TCPRoute or UDPRoute, which have different semantics.
• mTLS is enabled by default. The Mesh Controller issues short‑lived certificates (rotated every 24 h) and injects them into the proxy. If the controller is unavailable, certificates may expire and communication will fail.
• Because the proxy runs as a DaemonSet, a crash on a node affects all services on that node.

Worked Example: Deploying a Canary with Traefik Mesh

Below is a step‑by‑step guide to deploy two versions of a simple Go HTTP service and configure a 90/10 traffic split. The example assumes a Kubernetes cluster (e.g., kind or minikube) and Helm 3.0+.

1. Install Traefik Mesh

   helm repo add traefik-mesh https://helm.traefik.io/traefik-mesh
   helm repo update
   helm install traefik-mesh traefik-mesh/traefik-mesh --namespace traefik-mesh --create-namespace

   Run as a user with cluster‑admin privileges. Verify the DaemonSet is running:

   kubectl get ds -n traefik-mesh

2. Deploy two service versions

   # myapp-stable.yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: myapp-stable
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: myapp
         version: stable
     template:
       metadata:
         labels:
           app: myapp
           version: stable
       spec:
         containers:
         - name: server
           image: myorg/myapp:stable
           ports:
           - containerPort: 8080
   
   ---
   # myapp-canary.yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: myapp-canary
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: myapp
         version: canary
     template:
       metadata:
         labels:
           app: myapp
           version: canary
       spec:
         containers:
         - name: server
           image: myorg/myapp:canary
           ports:
           - containerPort: 8080

   Apply both manifests:

   kubectl apply -f myapp-stable.yaml
   kubectl apply -f myapp-canary.yaml

3. Create Services

   apiVersion: v1
   kind: Service
   metadata:
     name: myapp-stable
   spec:
     selector:
       app: myapp
       version: stable
     ports:
     - port: 80
       targetPort: 8080
   
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: myapp-canary
   spec:
     selector:
       app: myapp
       version: canary
     ports:
     - port: 80
       targetPort: 8080

   Apply:

   kubectl apply -f myapp-service.yaml

4. Define the HTTPRoute

   apiVersion: traefik.io/v1alpha1
   kind: HTTPRoute
   metadata:
     name: myapp-route
   spec:
     routes:
       - match: Host("myapp.example.com")
         services:
           - name: myapp-stable
             port: 80
             weight: 90
           - name: myapp-canary
             port: 80
             weight: 10

   Apply:

   kubectl apply -f myapp-route.yaml

5. Verify traffic distribution

   Expose the proxy via a LoadBalancer or use traefik-mesh CLI to forward traffic. Then send a burst of requests and inspect the X-Forwarded-For header or log output from each pod to confirm the 90/10 split. A simple way is to run:

   for i in {1..100}; do curl -s -o /dev/null -w "%{http_code}\n" http://myapp.example.com; done | sort | uniq -c

   Expected output roughly: 90 200 and 10 200, indicating the split.

6. Check mTLS health

   Run:

   kubectl exec -it myapp-stable-0 -n default -- sh -c "openssl s_client -connect $(hostname):80 -servername myapp.example.com"

   Look for a valid certificate signed by the Mesh CA. If the controller is down, the certificate might be expired.

Trade‑offs & Limitations

• Shared Data Plane: The DaemonSet proxy is a single process per node. A crash or misconfiguration can affect all services on that node, unlike sidecar models where each pod isolates its own proxy.
• Granularity of Network Policies: Because all traffic on a node shares the same proxy, you cannot enforce per‑pod network policies as tightly as with sidecar proxies.
• Non‑HTTP Traffic: HTTPRoute only handles HTTP/HTTPS. For TCP/UDP you need separate CRDs and the feature set is less mature.
• Limited Traffic Shaping: Advanced capabilities like fault injection, retries, or circuit breakers are not yet fully exposed through the HTTPRoute CRD.
• Certificate Rotation Dependence: The 24‑hour rotation interval works for most workloads, but if the Mesh Controller is down for an extended period, services may lose mTLS connectivity.

Actionable Takeaways

1. Use Traefik Mesh when you need a lightweight, low‑overhead way to perform canary releases on HTTP services.
2. Define traffic splits with HTTPRoute weights, keeping the sum of weights at 100 for clarity.
3. Monitor the DaemonSet’s health on each node; set up alerts for pod crashes or high CPU usage.
4. Validate mTLS by periodically checking the certificate validity period in application logs or via openssl.
5. If you require fine‑grained per‑pod policies or advanced traffic shaping, consider a sidecar‑based mesh or supplement Traefik Mesh with Kubernetes NetworkPolicies.

By following this pattern, teams can achieve rapid, safe canary deployments without the overhead of sidecar proxies, while staying aware of the trade‑offs inherent in a shared data plane.