Envoy control plane to data plane xDS v3: SotW omission versus Delta removed_resources semantics

Answer the question first

In Envoy v3, a resource that is omitted from a SotW snapshot will not be removed from the data plane if the data plane is still receiving a Delta stream for that type. The Delta stream will only delete a resource when it receives a removed_resources entry. Therefore, a SotW omission only takes effect on a type that is served exclusively via SotW.

Confirmed facts

• State‑of‑the‑world (SotW) streams delete resources by omission.
• Delta streams delete resources only when an explicit removed_resources message is sent.
• Mixed delivery (SotW for one type, Delta for another) can leave stale listeners, clusters, or routes if the removal semantics are not coordinated.
• TTL is optional; its effect on removal in mixed deployments is not fully documented.

Likely explanation for stale resources

If a resource is removed via SotW but the data plane still subscribes to a Delta stream for that type, the omission will be ignored. The resource remains until a removed_resources entry arrives. Conversely, a removal via Delta will be ignored by a SotW stream until the next full snapshot.

Practical guidance

1. Keep one stream type per resource: For each resource type, choose either SotW or Delta and stick to it. Mixing is only safe if you coordinate removal messages.
2. Explicitly remove omitted resources: If you must mix, whenever you omit a resource in a SotW snapshot, immediately publish a removed_resources message for that type on the Delta stream.
3. Use Envoy debug logs: Start Envoy with -v or --log-level debug to see when removed_resources entries are processed.
4. Verify with a test control plane: Serve the same resource type via both SotW and Delta in a staging environment and confirm that stale resources do not persist.

TTL considerations

TTL is optional. If a resource has a TTL, Envoy will drop it after the TTL expires only if the control plane does not override the behavior. In mixed deployments, a TTL expiry on a Delta stream does not automatically generate a removed_resources entry; the control plane should emit one if it wants the data plane to purge the resource immediately.

Migrating between SotW and Delta

1. Plan a cutover window: Disable the old stream type for the target resource type while you enable the new type.
2. Send a full snapshot via the new stream: Ensure the data plane receives a complete state before shutting down the old stream.
3. Remove orphaned resources: After the cutover, publish removed_resources entries for any resources that were present only in the old stream.

Diagnostic question

To fine‑tune the recommendation for your environment, could you confirm which Envoy major version you are running? Different releases have subtle bugs around mixed SotW/Delta handling.