Using Envoy’s Request Mirroring for Safe Canary Testing

Why Mirror Requests Instead of Splitting Traffic?

When you roll out a new API version, you want to see how it behaves under real traffic without risking the user experience. Traditional canary routing sends a fraction of traffic to the new service and relies on monitoring to catch issues. Request mirroring takes a different approach: every request that hits the primary service is duplicated and forwarded to a secondary, non‑blocking endpoint. The client never sees the mirrored traffic, so you can observe the new code in production conditions without affecting latency or availability.

Thesis: Mirroring Enables Risk‑Free Validation

Mirroring lets you validate the new service’s correctness, performance, and security in a real‑world context while keeping the user experience pristine. Because the mirrored request is processed by the target service, you get real logs, metrics, and side‑effects, but the client still receives the original response from the primary cluster.

Step‑by‑Step Implementation

1. Define the Staging Cluster

   Create a new cluster in Envoy’s static configuration that points to the staging or canary deployment of the API. For example:

   static_resources:
     clusters:
     - name: api-staging
       connect_timeout: 0.25s
       type: strict_dns
       lb_policy: round_robin
       hosts:
       - socket_address:
           address: api-staging.internal
           port_value: 8080
   

2. Add a Mirror Policy to the Route

   Within the HTTP connection manager’s route_config, add a mirror_policy block. You can mirror to a single cluster or multiple clusters, set a percentage, and filter headers.

   http:
     route_config:
       name: local_route
       virtual_hosts:
       - name: local_service
         domains: ["*"]
         routes:
         - match:
             prefix: "/"
           route:
             cluster: api-prod
             mirror_policy:
               cluster: api-staging
               mirror_percentage:
                 numerator: 100
                 denominator: HUNDRED
               request_headers_to_add:
               - header: "x-mirrored-by"
                 value: "envoy"
   

3. Deploy the Updated Envoy

   Reload Envoy with the new configuration. On Linux, you can use systemctl reload envoy or envoy -c /etc/envoy/envoy.yaml --service-node my-node --service-cluster my-cluster. Ensure you have sudo or the appropriate permissions.

4. Verify Mirrored Traffic

   Use Envoy’s stats endpoint to confirm that requests are being mirrored. Example using curl:

   curl http://localhost:9901/stats?format=prometheus | grep "cluster.api-staging.requests"
   

   The counter should increment in tandem with the primary traffic.

5. Inspect the Staging Service

   Check that the staging endpoint receives the same payloads. If you have access to the staging logs, you should see a request identical to the one the client made, minus any header filtering you applied.

Concrete Example: Mirroring a JSON POST to a Feature Flag Service

Suppose you’re adding a new feature flag endpoint /flags to your API. You want to see how the new implementation behaves under real traffic. Here’s a minimal configuration snippet that mirrors all POST requests to /flags to the staging cluster.

http:
  route_config:
    name: flag_route
    virtual_hosts:
    - name: flag_service
      domains: ["*"]
      routes:
      - match:
          prefix: "/flags"
          methods: ["POST"]
        route:
          cluster: api-prod
          mirror_policy:
            cluster: api-staging
            mirror_percentage:
              numerator: 50
              denominator: HUNDRED
            request_headers_to_add:
            - header: "x-mirrored"
              value: "true"

With mirror_percentage set to 50, half of the POST requests are duplicated. You can adjust this value to control load on the staging environment.

Trade‑offs and Limitations

• Resource Consumption – Mirrored requests still hit the staging service, consuming CPU, memory, and network bandwidth. Over‑mirroring can saturate a low‑capacity canary environment.

• Security Concerns – The full request body, including sensitive data, is sent to the staging endpoint. Use request_headers_to_add or body filtering (via Lua or HTTP filter) to redact confidential fields.

• Metric Skew – Traffic counters for the mirrored cluster will include mirrored requests. Filter or separate these metrics to avoid misinterpreting load.

• No Latency Impact on Clients – Because mirroring is non‑blocking, the client response time is unaffected. However, the mirrored service’s processing time may affect the overall resource usage of the Envoy instance.

Actionable Checklist for Production Deployment

1. Reserve a dedicated staging cluster with sufficient capacity.

2. Set mirror_percentage to a conservative value (e.g., 10–20%) during initial rollout.

3. Enable header filtering to remove or mask sensitive data before mirroring.

4. Configure separate Prometheus metrics or log streams for mirrored traffic to avoid conflating production and canary data.

5. Monitor Envoy’s /stats endpoint for mirrored request counts and any errors.

6. Once confidence is established, gradually increase mirror_percentage or remove mirroring after the feature is fully validated.

Conclusion

Envoy’s request mirroring offers a powerful, low‑risk way to validate new API changes against real traffic. By duplicating requests to a staging endpoint, you gain authentic observability without compromising user experience. Just remember to size the staging environment, protect sensitive data, and keep metrics separate. With a careful rollout, mirroring can become a staple in your continuous delivery pipeline.