Which mechanism ensures write idempotency during API server retries?

Question

When implementing custom controllers in Kubernetes (v1.28+), ensuring that retried write operations do not result in duplicate resources or overwritten data is critical. The API server utilizes optimistic concurrency control via the resourceVersion field to prevent conflicting updates during PUT requests.

While resourceVersion manages updates, the behavior for POST (create) operations relies on the uniqueness of the resource name within a namespace. However, in scenarios where a client receives a timeout or a network error after a write was successfully committed to etcd, a naive retry may result in a 409 Conflict if the resource already exists.

There is a design uncertainty regarding the most efficient way to handle these retries without introducing excessive GET requests to verify state before every attempt.

Does Server-Side Apply provide a more robust idempotency guarantee for retries compared to standard PUT operations?
  What is the recommended pattern for distinguishing between a legitimate conflict and a retry of a previously successful but unacknowledged write?

Tasadduq Burney · Answer

Idempotency‑Key header ensures write idempotency
The mechanism is a client‑generated Idempotency-Key header (or query parameter) that uniquely identifies a logical write operation.

Client generates a unique key (e.g., UUID) for each intended write.
  Key is sent in the Idempotency-Key header with the request.
  API server checks a persistent idempotency store for that key.
  If the key exists, the server returns the previously cached response without re‑executing the write.
  If the key is absent, the server processes the request, stores the key together with the response (or a completion marker), and returns the response.

This approach avoids extra GET checks and works for both POST creates and PUT updates.
# Example curl retry
curl -X POST https://api.example.com/resources \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: a3f9c2e1-7b4d-4a9b-8f6e-2d1c5b7a9f0e" \
  -d '{"name":"my-resource"}'

Which mechanism ensures write idempotency during API server retries?

1 answer

Idempotency‑Key header ensures write idempotency

Your perspective belongs here.

1 question comment

Verification Tip