Kubernetes API Server and client-go: Safe retry of POST creates without duplicate writes

Achieving Idempotency in Kubernetes POST Requests

The Kubernetes API server does not provide native idempotency keys or request IDs for POST operations. Because POST is non-idempotent by design, a retry after a network timeout or a 503 Service Unavailable response can result in duplicate resources or, more commonly, a 409 Conflict error if the first request actually succeeded.

Recommended Implementation Strategies

To prevent duplicate writes and safely handle retries, you should move away from raw POST requests in favor of one of the following patterns:

1. Server-Side Apply (SSA)

The most robust way to ensure idempotency is using Server-Side Apply via a PATCH request. SSA allows you to send the desired state of the object; if the object does not exist, the API server creates it. If it does exist, it updates it. This operation is idempotent by nature.

• Verb: PATCH
• Content-Type: application/apply-patch+yaml
• Requirement: You must specify a fieldManager in the request to track ownership of the fields.

2. Deterministic Naming with Conflict Handling

If you must use POST, you must ensure the resource name is deterministic (not generated by the server via generateName). You can then wrap your client-go logic in a retry loop that explicitly handles the 409 Conflict status code.

// Conceptual client-go logic for idempotent create
err := retry.OnError(wait.Backoff{...}, func(err error) bool {
    return errors.IsConflict(err) || errors.IsServiceUnavailable(err)
}, func() error {
    _, err := client.CoreV1().Pods(ns).Create(ctx, pod, metav1.CreateOptions{})
    if errors.IsAlreadyExists(err) {
        // The resource was created by a previous attempt that timed out
        return nil 
    }
    return err
})

Analysis of 503 and Retry-After

A 503 Service Unavailable response is inherently ambiguous. It may be triggered by a load balancer before the request reaches the API server, or by the API server after the write has been committed to etcd but before the response was dispatched. There is no header or signal in the 503 response that confirms whether the write was processed.

Summary of Mechanisms

Mechanism	Idempotent?	Behavior on Retry
POST (Random Name)	No	Creates duplicate resources.
POST (Fixed Name)	Partial	Returns 409 Conflict if already exists.
PATCH (SSA)	Yes	Ensures desired state regardless of existing object.

Missing Diagnostic: To refine this recommendation, please specify if you are using generateName for your resources, as this precludes the use of 409 Conflict detection.