Adding Retry Middleware to Traefik Ingress for Resilient Kubernetes Services

Problem: intermittent backend failures cause visible errors

In a Kubernetes cluster, a short‑lived pod restart or a temporary network glitch can turn a successful HTTP request into a 5xx error. Applications that rely on the ingress layer see these failures propagate to users, even though the underlying service recovers within seconds.

Thesis: Traefik’s retry middleware can mask short‑lived failures

By inserting a retry policy into the Traefik middleware chain, failed requests are automatically re‑routed according to a configurable attempt count and back‑off schedule. This improves perceived reliability without changing application code.

How the retry middleware works

The retry middleware sits between the router and the service. When Traefik receives a response that matches the retry conditions (by default, 5xx status codes or connection failures), it re‑issues the request up to attempts times, waiting initialInterval multiplied by 2^(n‑1) milliseconds between attempts (exponential back‑off). The middleware can be scoped to a specific router, service, or applied globally via a dynamic file or Kubernetes CRD.

Worked example: retry a simple Go service

1. Deploy a test service (run as a user with edit rights in the target namespace):

   kubectl apply -n demo -f - <<'EOF'
   apiVersion: v1
   kind: Service
   metadata:
     name: retry-demo
   spec:
     ports:
     - port: 8080
       targetPort: 8080
     selector:
       app: retry-demo
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: retry-demo
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: retry-demo
     template:
       metadata:
         labels:
           app: retry-demo
       spec:
         containers:
         - name: app
           image: golang:1.22-alpine
           command: ["/bin/sh","-c"]
           args:
             - |
               mkdir -p /app
               cat > /app/server.go <<'GO'
   package main
   import (
   	"fmt"
   	"net/http"
   	"time"
   )
   var counter int
   func handler(w http.ResponseWriter, r *http.Request) {
   	counter++
   	if counter <= 2 {
   		// fail first two requests
   		w.WriteHeader(http.StatusInternalServerError)
   		fmt.Fprintln(w, "temporary failure")
   		return
   	}
   	fmt.Fprintf(w, "success after %d attempts\n", counter)
   }
   func main() {
   	http.HandleFunc("/", handler)
   	http.ListenAndServe(":8080", nil)
   }
   GO
               go run /app/server.go
           ports:
           - containerPort: 8080
   EOF
   

2. Define the retry middleware as a Traefik Middleware CRD:

   kubectl apply -n demo -f - <<'EOF'
   apiVersion: traefik.io/v1alpha1
   kind: Middleware
   metadata:
     name: retry-demo
   spec:
     retry:
       attempts: 3
       initialInterval: 500ms
       # retry on 5xx and connection errors (default)
   EOF
   

3. Create an IngressRoute that wires the router, service, and middleware:

   kubectl apply -n demo -f - <<'EOF'
   apiVersion: traefik.io/v1alpha1
   kind: IngressRoute
   metadata:
     name: retry-demo-route
   spec:
     entryPoints:
       - web
     routes:
     - match: Host(`retry-demo.example.com`)
       kind: Rule
       services:
       - name: retry-demo
         port: 8080
       middlewares:
       - name: retry-demo
   EOF
   

4. Generate traffic and observe retries. From a pod or your laptop with hey installed:

   hey -z 10s -c 5 http://retry-demo.example.com/
   

   While the test runs, kill the backend pod to simulate a failure:

   kubectl delete pod -n demo -l app=retry-demo
   

   After a few seconds, redeploy the pod (the Deployment will recreate it). Check Traefik access logs (if enabled) for lines showing the same request being logged multiple times; the count should match attempts (3) before a successful response is returned.

Trade‑offs and limitations

• Added latency: each retry adds the back‑off delay. With 3 attempts and 500 ms initial interval, worst‑case extra latency is 500 ms + 1 s = 1.5 s.
• Load amplification: during a sustained outage, each client request may generate multiple backend hits, increasing pressure on the failing service.
• Idempotency risk: retrying non‑idempotent endpoints (e.g., POST that creates a resource) can cause unintended side effects.

Mitigation strategies:

• Pair retry with a circuitbreaker middleware to stop forwarding requests after a failure threshold.
• Enable rateLimit middleware to cap the amplified traffic.
• Apply retry only to routes known to be idempotent (GET, HEAD, safe PUT/DELETE).

Actionable closing

If your Kubernetes workloads experience brief, recoverable failures, start by adding a scoped retry middleware to the relevant IngressRoute. Verify the behavior with a load generator and pod‑kill test, monitor latency and request counts in Traefik logs, and then consider adding a circuit‑breaker or rate‑limiter to guard against load spikes. This incremental approach gives you measurable resilience gains without a major code change.