Zero-Downtime Config Updates with Envoy xDS: How CDS, EDS, LDS, and RDS Work Together

The Problem: Restarting Proxies to Change Routing

Every team running Envoy at scale hits the same wall: you need to shift traffic to a new service version, add a retry policy, or rotate a TLS certificate, and the only way to apply the change is editing a static YAML file and restarting the proxy. Even a graceful reload drops in-flight requests and adds latency spikes. Multiply that by hundreds of sidecars and you have a coordination nightmare.

Envoy's xDS (Universal Data Plane API) solves this by splitting configuration into four independently versioned resource types that a control plane can push over gRPC or REST — no proxy restart required. The proxies acknowledge each update with a version hash, giving you eventual consistency and a safe rollback path if a bad config lands.

Four Resource Types, One Subscription Model

xDS separates concerns so each team owns what they understand:

• LDS (Listener Discovery Service) — network filters, TLS contexts, and socket addresses. Changing a listener's filter chain (e.g., adding an auth filter) only touches LDS.
• RDS (Route Discovery Service) — HTTP route tables: virtual hosts, match conditions, retries, timeouts, and weighted clusters. You can rewrite routing logic without touching upstream definitions.
• CDS (Cluster Discovery Service) — upstream cluster definitions: load balancing policy, circuit breakers, outlier detection, and TLS verification. Cluster identity stays stable while endpoints change.
• EDS (Endpoint Discovery Service) — the actual IP:port endpoints for each cluster, plus metadata like locality, health status, and load balancing weight.

Each resource carries an opaque version string (typically a SHA-256 of the protobuf). Envoy rejects any update with a stale version, preventing split-brain configs during control-plane leader elections. The control plane pushes incremental updates; the proxy ACKs on success, NACKs on failure, and the control plane can retry or roll back.

ADS: Atomic Multi-Resource Updates

Running four separate gRPC streams works, but it creates a window where a listener might point to a route that references a cluster not yet updated. Aggregated Discovery Service (ADS) multiplexes LDS, RDS, CDS, and EDS over a single gRPC connection and applies them atomically. The control plane sends a single DeltaDiscoveryResponse (or SotW DiscoveryResponse) containing all four resource types; Envoy applies them as a unit or rolls back the entire set.

ADS is the default for production control planes (Istio, Consul Connect, custom Go-control-plane). If you're building your own, use the ADS endpoint — it eliminates the "listener points to missing route" race condition.

Delta xDS: Bandwidth Savings for Large Fleets

State of the World (SotW) sends the full resource list on every change. For a fleet with 10,000 proxies and 500 clusters, that's a lot of repeated protobuf. Delta xDS (Envoy 1.14+) sends only the resources that changed, keyed by their version hash. The control plane must track per-resource versions and support DeltaDiscoveryRequest/Response; Envoy falls back to SotW if the control plane doesn't implement delta.

Delta xDS cuts bandwidth roughly proportionally to the change rate. A single endpoint weight adjustment in a 500-endpoint cluster becomes a few KB instead of the full EDS payload. The trade-off: your control plane needs a version store (etcd, Redis, in-memory with persistence) and must handle resource deletion notifications explicitly.

Worked Example: Canary Deployments via EDS Weight Adjustment

Suppose you run payment-service v1.2 across two Kubernetes clusters (us-east, us-west). You want to shift 10% of traffic to v1.3 in us-east only, validate error rates, then ramp to 100%. With static config you'd edit the cluster definition and reload. With xDS you only touch EDS.

# Control plane pushes this EDS update (protobuf text format for clarity)
endpoints:
- locality: { region: "us-east", zone: "us-east-1a" }
  lb_endpoints:
  - endpoint:
      address: { socket_address: { address: "10.0.1.12", port_value: 8080 } }
    load_balancing_weight: 90   # v1.2
    metadata:
      filter_metadata:
        envoy.lb:
          version: "v1.2"
  - endpoint:
      address: { socket_address: { address: "10.0.1.45", port_value: 8080 } }
    load_balancing_weight: 10   # v1.3 canary
    metadata:
      filter_metadata:
        envoy.lb:
          version: "v1.3"
- locality: { region: "us-west", zone: "us-west-2b" }
  lb_endpoints:
  - endpoint:
      address: { socket_address: { address: "10.1.2.30", port_value: 8080 } }
    load_balancing_weight: 100  # v1.2 only in us-west
    metadata:
      filter_metadata:
        envoy.lb:
          version: "v1.2"

The control plane sends only the changed EDS resource (Delta xDS) or the full EDS set (SotW). Envoy applies the new weights immediately — in-flight requests complete on the old endpoints, new requests respect the updated weights. No RDS, CDS, or LDS changes needed. To roll back, push the previous EDS version; Envoy's version hash check ensures it accepts the older config.

Verification: On any proxy, hit the admin endpoint GET /config_dump?resource=eds_config_dump (requires admin interface enabled on 127.0.0.1:9901 by default). Confirm the version_info matches the hash your control plane sent and that load_balancing_weight values reflect the canary split.

Trade-offs and Operational Gotchas

• API version mismatch: Envoy 1.18+ defaults to xDS v3. If your control plane speaks v2 (pre-1.18), proxies will reject configs. Run envoy --version and confirm the control plane's gRPC API matches.
• Health check flapping: EDS endpoints carry health status from active checks (HTTP/TCP) or passive outlier detection. Aggressive intervals (< 5s) or tight thresholds cause endpoints to flap, triggering constant EDS updates. Tune interval, timeout, unhealthy_threshold, healthy_threshold to your service's SLA — start with 10s/5s/3/2 for HTTP.
• Thundering herd on rollout: Pushing a config change to 10k+ proxies simultaneously can overwhelm the control plane. Use incremental push strategies: shard the fleet, add jittered backoff, or leverage Delta xDS so only affected proxies receive updates.
• SDS is separate: TLS certificate rotation uses Secret Discovery Service (SDS), not xDS. Coordinate SDS and CDS/EDS updates so a new certificate arrives before the cluster starts using it — otherwise you get cert/key mismatches.

How to Verify xDS Is Working in Your Cluster

1. Confirm xDS v3 support: envoy --version (run on the proxy host; requires Envoy binary in PATH). Output should show 1.18 or later.
2. Inspect applied resources: curl -s http://127.0.0.1:9901/config_dump | jq '.configs[] | select(.@type | contains("eds")) | .version_info' (requires admin port access and jq). Each resource's version_info should match the control plane's last push hash.
3. Watch subscription logs: envoy -l debug --config-path /etc/envoy/envoy.yaml 2>&1 | grep -E 'subscription|ACK|NACK' (run in foreground or check log file). Look for successful ACK cycles and zero NACKs after a config push.
4. Test Delta xDS: Push a single endpoint weight change via your control plane, then re-run step 2. Only the EDS resource's version_info should change; LDS/RDS/CDS hashes stay identical.

If any check fails, start with the control plane's xDS gRPC logs — most issues are version mismatches, missing ADS muxing, or health check misconfiguration.

Next Steps

If you're on static config today, pick one cluster and migrate its endpoints to EDS first — it's the lowest-risk entry point. Pair Envoy with a control plane that implements ADS and Delta xDS (Istio, Consul Connect, or a thin Go-control-plane wrapper). Instrument the admin endpoint checks above into your deployment pipeline so every rollout verifies xDS convergence before declaring success.