Architecture Note: Helm Chart Hooks – Requirements, Minimal Design, Trust Boundaries, and Failure Modes

Problem and Takeaway

When installing a Helm chart you may need to run a one‑time task—such as creating a secret, running a database migration, or validating prerequisites—before the main workloads are created. If the task fails you want the whole release to abort and you do not want stray resources left behind. Helm Chart Hooks provide this capability, but they introduce trust boundaries and operational considerations that must be understood before relying on them.

Takeaway: Use a hook only when the task must run at a precise lifecycle point, scope its permissions to the release namespace (or grant explicit cluster‑wide RBAC), and always set a delete policy so that successful hooks are cleaned up automatically.

Requirements

To use Helm Chart Hooks you need:

• Helm 3 (the Tiller‑less client). The hook runs with the same service account that the helm command uses.
• Permission to create the hook resource kind (usually a Job or Pod) in the target namespace.
• If the hook needs to talk to the Kubernetes API, a service account with appropriate RBAC roles bound in the namespace (or cluster‑wide roles for cluster‑scoped resources).
• The hook manifest must contain the annotation helm.sh/hook with a valid lifecycle value (e.g., pre-install, post-install, pre-delete, post-delete, test).

Minimal Design

The smallest functional hook is a Kubernetes Job that prints a message and exits cleanly. Example manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "chart.fullname" . }}-pre-install
  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: hook
        image: busybox:1.36
        command: ["sh", "-c", "echo 'pre‑install hook running' && sleep 2"]

The helm.sh/hook-delete-policy: hook-succeeded tells Helm to delete the Job after it completes successfully. If you omit the delete policy, the Job remains after a successful release, which can clutter the namespace.

Trust and Data Boundaries

Hooks inherit the privileges of the Helm client’s service account. By default this account is limited to the namespace where the release is installed, unless you explicitly bind a ClusterRole or give the service account additional namespace roles. Therefore:

• A hook that only creates namespaced resources (ConfigMap, Secret, Job, Pod) stays within the release’s trust boundary.
• A hook that attempts to create cluster‑scoped resources (ClusterRole, Namespace, CRD) requires cluster‑wide RBAC. Granting such privileges widens the trust boundary and can affect other releases or workloads in the cluster.
• If the hook needs to read secrets from another namespace, you must add a role binding that permits cross‑namespace access; otherwise the hook will fail with an authorization error.

Operational Checks

After a helm install/upgrade/delete you should verify the hook’s outcome:

1. List hook resources: kubectl get jobs -n <namespace> -l helm.sh/hook=pre-install (adjust label or name as needed).
2. Check the Job’s completion status: kubectl describe job <hook-name> -n <namespace> looks for Succeeded: 1 or Failed: 1.
3. If successful, confirm the resource was removed according to its delete policy: kubectl get job <hook-name> -n <namespace> should return NotFound for hook-succeeded.
4. If the hook failed, inspect its logs: kubectl logs job/<hook-name> -n <namespace> to see the container output.
5. Finally, verify the Helm release status: helm status <release-name> -n <namespace> shows failed if any hook returned a non‑zero exit code.

Failure Modes

Typical ways a hook can cause problems:

• Non‑zero exit code: Helm treats this as a hook failure, aborts the release, and leaves any previously created hook resources unless a delete policy (e.g., hook-failed) is set.
• Missing delete policy: Successful hooks linger as Jobs, which can be mistaken for stale resources and may cause name collisions on subsequent releases.
• Cluster‑wide side effects: A hook that creates a ClusterRole without proper scoping can affect other namespaces, violating isolation assumptions.
• Insufficient permissions: The hook fails with an authorization error, causing the release to fail even though the chart’s main templates are fine.
• Long‑running hook: If the hook does not complete (e.g., waits for an external system), Helm will wait until the hook’s deadline (default 5 minutes) and then mark the release as failed.

Conditions That Would Change the Design

You would reconsider using a hook if any of the following apply:

• The task can be expressed as a regular container sidecar or init container within the application pod, eliminating the need for a separate lifecycle hook.
• You need guaranteed execution even when a release is rolled back; hooks do not run on rollback unless you explicitly define a post-rollback hook.
• Operational policy forbids granting the release’s service account any ability to create Jobs or Pods; in that case you must use an external controller or operator.
• The task must run only once per cluster (e.g., installing a CRD) and you want to avoid re‑running it on every namespace install; a hook would run per release, so a separate bootstrap step is preferable.
• You require detailed audit trails that Helm does not provide; an external CI/CD pipeline with its own logging and approval steps would be more suitable.

Concrete Example: Pre‑Install Secret Creation

Suppose a chart needs a secret containing a database password before the main Deployment starts. The hook creates the secret, and the Deployment references it via secretRef. The hook manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-db-secret
  annotations:
    "helm.sh/hook": pre-install
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      restartPolicy: OnFailure
      serviceAccountName: {{ .Values.serviceAccountName }}
      containers:
      - name: create-secret
        image: bitnami/kubectl:latest
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: {{ .Release.Name }}-db-cred
              key: password
        command:
        - sh
        - -c
        - |
          kubectl create secret generic {{ .Release.Name }}-db-cred \
            --from-literal=password="$(openssl rand -base64 32)" \
            --dry-run=client -o yaml | kubectl apply -f -

To try this:

1. Save the hook as templates/secret-hook.yaml in a chart.
2. Ensure the chart’s values.yaml defines a service account that has role allowing secrets creation in the namespace (e.g., a role with secrets verbs).
3. Run helm install myrelease ./mychart -n testns with a user that can create Jobs and the service account.
4. After install, run kubectl get job -n testns -l helm.sh/hook=pre-install to see the hook Job.
5. Check its logs: kubectl logs job/myrelease-db-secret -n testns.
6. Verify the secret exists: kubectl get secret myrelease-db-secret -n testns -o yaml.
7. Confirm the Job was removed: kubectl get job myrelease-db-secret -n testns should return NotFound.
8. Check the release status: helm status myrelease -n testns shows deployed.

If you modify the hook to exit with exit 1, the install will abort, the hook Job will remain (unless you added hook-failed to the delete policy), and helm status will report failed.

Limitations and Practical Verification

Hooks are not a substitute for proper application startup logic. They add operational surface area:

• They increase the number of objects that must be monitored and garbage‑collected.
• They can mask errors in the main chart if the hook succeeds but the application still fails to start.
• Debugging hook failures requires looking at Job logs, which are separate from Helm’s output.

A practical way to confirm that a hook behaves as expected in your environment is to:

1. Deploy a test chart with a hook that writes a known file to a ConfigMap.
2. After install, check that the ConfigMap contains the expected data.
3. Then deliberately break the hook (e.g., change the command to exit 1) and reinstall; verify that the release is marked failed and that no main resources were created.
4. Finally, restore the hook and ensure the release succeeds and the hook is cleaned up according to its delete policy.

By following these steps you gain confidence that the hook’s trust boundary, permissions, and cleanup behavior match your operational assumptions.