Diagnosing Helm 3 Upgrade Failures Caused by Immutable Fields

The Problem: API Server Rejection

A helm upgrade command fails with a non-zero exit code, and the error message explicitly mentions that a field is immutable. In some cases, the command may return, but the release status remains stuck in pending-upgrade or pending-install.

The core issue is that Kubernetes prevents certain fields from being updated after a resource is created to maintain system stability. When Helm attempts to patch a resource with a new value for one of these fields, the Kubernetes API server rejects the entire request. Helm cannot "force" this change because it is a restriction of the Kubernetes API, not a limitation of Helm itself.

Cause and Diagnostic Mapping

Observed Symptom	Likely Immutable Cause	Technical Reason
Error mentions spec.clusterIP	Service type change or IP modification	Service IP allocations are fixed upon creation.
Error mentions spec.selector	Deployment or StatefulSet label change	Selectors define the relationship between controllers and pods; changing them would orphan existing pods.
Error mentions spec.template.metadata.labels	Pod template label mismatch	Labels must align with the immutable selector to be valid.
Error mentions container ports	Port name or protocol change	Certain container port definitions are immutable depending on the K8s version.

Ordered Diagnostic Checks

Perform these checks in sequence to isolate exactly which field is causing the conflict.

1. Identify the Failed Revision

Run this command from a workstation with helm and kubectl access. You must have RBAC permissions to read Secrets in the release namespace, as Helm 3 stores history there.

helm history <release-name> -n <namespace>

Locate the revision marked as FAILED. Note the specific error message in the DESCRIPTION column. If the history is empty but you know the release exists, verify your RBAC permissions for Secrets.

2. Diff Rendered Manifests

Compare what Helm tried to deploy against what was previously successful. This identifies the change introduced in the chart or values.yaml.

# Save the last successful manifest
helm get manifest <release-name> <last-success-rev> -n <namespace> > good.yaml

# Save the failed manifest
helm get manifest <release-name> <failed-rev> -n <namespace> > bad.yaml

# Compare the two
diff -u good.yaml bad.yaml

3. Compare Live Object to Manifest

Verify if the live object in the cluster matches the last successful manifest or if manual "drift" has occurred.

kubectl get <resource-type> <resource-name> -n <namespace> -o yaml > live.yaml
diff -u good.yaml live.yaml

Remediation Paths

Path A: Revert the Value (No Change Required)

If the change to the immutable field was accidental or unnecessary, revert the value in your values.yaml or chart template to match the live.yaml output, then run the upgrade again.

helm upgrade <release-name> <chart> -n <namespace> -f values.yaml

Path B: Rollback for Immediate Recovery

If the environment is unstable and you need to return to a known good state quickly.

helm rollback <release-name> <last-success-rev> -n <namespace>

Risk: Rollback reverts the Helm manifest but does not undo manual kubectl patch or edit operations performed outside of Helm.

Path C: Controlled Resource Replacement

When the immutable field must change, the resource must be deleted and recreated. This will cause a brief service interruption.

1. Rollback the release to the last successful version to clear the pending-upgrade state.
2. Manually delete the offending resource: kubectl delete <type> <name> -n <namespace>.
3. Run helm upgrade again. Helm will see the resource is missing and create it fresh with the new immutable values.

Caution: For stateful workloads (e.g., StatefulSets), ensure you do not delete the underlying PersistentVolumeClaims (PVCs) unless data loss is acceptable.

Verification and Limitations

After applying a fix, verify the result with these checks:

• Run helm status <release-name> -n <namespace>; the status should be deployed.
• Run helm history <release-name> -n <namespace> to ensure the latest revision is no longer FAILED.
• Compare helm get manifest with kubectl get -o yaml to ensure the immutable field now matches.

Critical Limitation: Never manually edit a live resource to "trick" Helm into thinking the field matches. This creates configuration drift; the next time you run an upgrade, Helm will attempt to apply the chart's value again, and the upgrade will fail once more.

Escalation Criteria

Escalate to the platform or cluster administration team if:

• helm rollback fails or the release remains in a pending state despite remediation.
• The resource is a critical stateful component where deletion would cause permanent data loss.
• The same immutable field error appears across multiple different releases, suggesting a systemic bug in a shared library chart.