Why does an Envoy listener route work locally but fail in production?

Question

When deploying the same Envoy configuration that passes local integration tests, the listener fails to match a specific HTTP route in the production environment, returning a 404 response despite identical YAML files.

The setup uses a static listener with a route configuration that includes a prefix matcher on /api/ and a corresponding cluster. No dynamic discovery services are involved, and the Envoy binary version matches the one used locally.

Possible causes could involve differences in listener filters, runtime overrides, or subtle variations in the effective configuration after hot‑restart. Identifying the exact point where the route table diverges is unclear.

What factors might cause the route to be ignored only in prod? How can differences in filter chains or runtime settings affect matching? Which Envoy admin endpoints or logs help diagnose the mismatch?

Wasi · Answer

The Primary Cause: Request Transformation

When identical Envoy YAML configurations behave differently across environments, the issue is rarely the configuration file itself but the effective request that reaches the Envoy listener. In production, Envoy is typically positioned behind a Load Balancer (LB) or Ingress controller, which can modify the HTTP request in ways that break prefix matching or header‑based routing.

Likely Explanations vs. Confirmed Facts

While the YAML is identical, the environment context differs. Consider these likely causes:

Header Stripping/Modification: Production LBs may strip or rename headers that your route depends on, or they may normalize paths (e.g., removing trailing slashes) before the request reaches Envoy.
TLS Termination Mismatches: If production uses HTTPS termination at the LB and forwards HTTP to Envoy, but the listener expects specific ALPN or SNI settings, the request may be routed to a default listener rather than the intended one.
Path Rewriting: An upstream proxy might rewrite /api/resource to /resource before it hits Envoy, causing a prefix matcher for /api/ to fail.

Diagnostic Steps

To isolate where the route table diverges, use these scoped verification methods:

Dump the Active Configuration: Compare the actual runtime state, not the source file. Use the admin endpoint to see what Envoy has actually loaded:
```
curl http://localhost:8001/config_dump
```
Inspect the Request Flow: Use curl -v against the production endpoint. Check if the 404 is coming from Envoy (look for the server: envoy header) or from the upstream Load Balancer.
Analyze Access Logs: Enable access_log in your listener. Check the %RESPONSE_CODE% and %UPSTREAM_CLUSTER%. If the cluster is -, Envoy failed to match the route; if a cluster is listed but it returns 404, the issue is with the backend service.

Missing Diagnostic Detail

To provide a more specific recommendation: Is the 404 response containing the server: envoy header? If it is missing, the request is failing at the Load Balancer before it ever reaches your Envoy listener.

Why does an Envoy listener route work locally but fail in production?

1 answer

The Primary Cause: Request Transformation

Likely Explanations vs. Confirmed Facts

Diagnostic Steps

Missing Diagnostic Detail

Your perspective belongs here.

1 question comment

Why a Listener Might Be Ignored in Production