Managing Kubernetes Resource Drift with Helm 3 Three-Way Strategic Merge

The Problem: Overwriting Manual Cluster Changes

When upgrading a Helm release, a standard two-way diff (comparing the old manifest to the new one) often clobbers changes made to resources directly in the cluster via kubectl edit or external controllers. This leads to "flapping" configurations where manual hotfixes are wiped out every time a CI/CD pipeline triggers a Helm upgrade.

The solution is the Three-Way Strategic Merge Patch. Instead of just comparing two files, Helm 3.2+ compares the live cluster state, the previously deployed manifest, and the new desired manifest. This allows Helm to distinguish between a field that was intentionally removed in the new chart and a field that was added manually to the live resource, preserving the latter where possible.

Prerequisites

• Helm Version: 3.2 or higher (where three-way merge became the default).
• Cluster Access: RBAC permissions to read and write resources in the target namespace.
• State: An existing Helm release already deployed to the cluster.

How the Three-Way Merge Works

Helm computes the final state by analyzing three distinct data points:

1. Old Manifest: The state of the resources as they were during the last successful helm upgrade (stored in a Kubernetes Secret sh.helm.release.v1...).
2. Live State: The current actual state of the resource in the Kubernetes API.
3. New Manifest: The rendered templates from the updated chart and values file.

If a field exists in the Live State but was not present in the Old Manifest, Helm assumes it was added externally and preserves it. If a field was present in the Old Manifest but is missing in the New Manifest, Helm explicitly removes it from the Live State.

Practical Upgrade Procedure

To perform an upgrade while leveraging the three-way merge, follow these steps. We will use a hypothetical release named web-app.

1. Audit the Current State

Before upgrading, capture the manifest Helm believes is currently deployed to compare it against the live API state.

# Run on local machine with cluster access
helm get manifest web-app -n production > old_manifest.yaml
kubectl get deployment web-app-deployment -n production -o yaml > live_state.yaml

2. Preview the New Manifest

Render the new chart version to a file to see exactly what the "New Manifest" will look like before it hits the cluster.

# Render the new version without installing
helm template web-app ./charts/web-app --version 1.2.0 -f values.yaml > new_manifest.yaml

3. Execute the Upgrade

Run the upgrade command. Helm will automatically perform the three-way merge in the background.

# Perform the upgrade
helm upgrade web-app ./charts/web-app -n production --version 1.2.0 -f values.yaml --atomic

Note: The --atomic flag is recommended; it automatically rolls back the release if the upgrade fails to reach a ready state.

Verification and Diagnostics

To verify that the merge respected field ownership, inspect the managedFields of the resource. Kubernetes uses these fields to track which actor (Helm, kubectl, or a controller) modified specific properties.

# Check field ownership on a specific resource
kubectl get deployment web-app-deployment -n production -o yaml | grep -A 10 "managedFields:"

If the merge worked, you should see manager: helm associated with the fields defined in your chart, while manual edits may be attributed to manager: kubectl-edit.

Comparison: Two-Way vs. Three-Way Merge

Scenario	Two-Way Merge (Legacy)	Three-Way Merge (Helm 3.2+)
Manual label added to Pod	Overwritten/Removed during upgrade	Preserved (not in old manifest)
Field removed from Chart	Ignored (remains in cluster)	Removed from cluster
Value changed in Chart	Updated to new value	Updated to new value

Limitations and Risks

• Immutable Fields: The three-way merge cannot bypass Kubernetes API restrictions. Changing immutable fields (e.g., spec.clusterIP of a Service) will cause the upgrade to fail. You must delete the resource manually or use --force (which deletes and recreates the resource).
• CRD Schema Requirements: The strategic merge relies on OpenAPI v3 schemas. If you are using Custom Resource Definitions (CRDs) that lack a proper schema, Helm falls back to a JSON Merge Patch, which replaces the entire object rather than merging individual fields.
• Secret Size Limits: Because Helm stores the old manifest in a Kubernetes Secret, extremely large releases (approaching 1MB) may encounter storage errors, potentially breaking the merge logic.

Recovery Options

If the upgrade results in an unexpected state, use the rollback command to return to a previous revision. This operation also uses a three-way merge, treating the target revision as the "New Manifest."

# Rollback to revision 1
helm rollback web-app 1 -n production