Gradual Canary Releases with Traefik Mesh Weighted Routing on Docker Swarm

The problem: cutovers are binary inside a Swarm

You deploy a new version of a service to Docker Swarm mode, but the old and new containers both need to serve live traffic for a while. A global update flips every replica at once, which means a bug in the new image hits all users immediately. Traefik Mesh offers a middle ground: weighted traffic splitting between two Swarm services that share a logical mesh name, so you can send a small percentage of intra-cluster requests to a canary backend and raise it gradually.

The useful takeaway: declare traffic shares with Swarm service labels, let the Mesh control plane regenerate the Traefik data plane configuration dynamically, and verify the split by calling the mesh service name from another Swarm service. This only works inside the cluster; external users still need a separate Traefik Proxy ingress route.

How weighted labels map to backends

Traefik Mesh watches the Swarm API and turns service labels into dynamic routing configuration. Weighted splitting uses the label prefix traefik.http.services., followed by a service name you choose, then .weighted.services., a backend identifier, and .weight. The weight is a relative integer, not a percentage: two backends with weights 90 and 10 split traffic roughly 90/10.

Both versions must be discoverable under the same mesh service name that clients use. The Mesh control plane propagates label changes without restarting the proxy, so you can adjust weights while the stack is running.

Worked example: shift 10 percent to a canary

Run these commands on a Swarm manager node. You need access to the Docker socket, typically through the docker group or sudo. Replace placeholders in angle brackets with your own image tags and service names.

First, create the stable backend:

docker service create \
  --name api-stable \
  --label traefik.enable=true \
  --label "traefik.http.services.api.weighted.services.stable.weight=90" \
  --label "traefik.http.services.api.weighted.services.stable.loadBalancer.server.port=8080" \
  :1.0.0

Then create the canary backend:

docker service create \
  --name api-canary \
  --label traefik.enable=true \
  --label "traefik.http.services.api.weighted.services.canary.weight=10" \
  --label "traefik.http.services.api.weighted.services.canary.loadBalancer.server.port=8080" \
  :2.0.0-rc1

Clients inside the mesh call api (the logical service name in the weighted block), not api-stable or api-canary directly. To move traffic, update the weights on the existing services:

docker service update \
  --label-add "traefik.http.services.api.weighted.services.stable.weight=70" \
  --label-add "traefik.http.services.api.weighted.services.canary.weight=30" \
  api-stable

Because labels are attached to individual services, you may need to run the matching --label-add against both service IDs if your labeling scheme stores the weighted block on each replica. Confirm syntax before relying on the split:

docker service inspect api-stable --format '{{json .Spec.Labels}}'

Expected check: the JSON output should contain the weighted label keys with integer values. A typo in the key path can silently collapse all traffic to one backend, so inspect before and after every change.

Verify the split from inside the cluster

Run a temporary debug container on the same Swarm network and request the mesh service name repeatedly:

docker run --rm --network  appropriate/curl -s -H "Host: api" http://api:8080/version

If your applications emit an X-Backend-Version header or similar, count the responses over several dozen requests. Roughly one in ten should identify the canary at the 10/90 starting point. Do not use curl from your laptop: external traffic does not traverse Mesh weights unless you have configured a separate Traefik Proxy entrypoint and route.

Limitations to plan around

• Swarm only. Weighted Mesh routing is scoped to Docker Swarm mode. Kubernetes and standalone Compose projects use different Traefik routing mechanisms.
• No connection draining. Changing a weight affects new requests, but in-flight or long-lived connections can finish on the previous backend until they close.
• Silent misconfiguration. An invalid label path does not always produce a clear error; it may simply route 100 percent of traffic to one backend.
• Maintenance status. Traefik Mesh is not Traefik's current primary product focus. Confirm active support and compatibility with your Docker Engine version before production use.

Rollback and cutover

Changing weights alters cluster state, so have a reverse operation ready. To abort a canary, set the canary weight to 0 and the stable weight to 100 with docker service update --label-add, then remove the canary service once traffic has fully shifted. Existing connections may still complete on the canary until they end.

Practical closing checklist

1. Label both stable and canary services under the same weighted service name.
2. Inspect labels with docker service inspect before and after changes.
3. Verify distribution from a Swarm client using response headers such as X-Backend-Version.
4. Confirm your Docker Engine and Traefik Mesh versions are compatible before upgrading the cluster.

Weighted Mesh routing is a practical way to reduce blast radius for Swarm workloads, but it is not a substitute for observability. Check the generated dynamic configuration in the Traefik Mesh dashboard or API after the first weight change, and treat the initial rollout as an experiment until the header counts match your target split.