Running Database Migrations Safely with Helm Hooks

The problem: migrations that run too late or too often

When you deploy a service that depends on a database, schema changes must be applied before the new containers start. If you run the migration manually or as a side‑car, you risk:

• Starting the application while the old schema is still in place, causing errors.
• Running the migration multiple times during a single upgrade, which can corrupt data if the script isn’t idempotent.

You need a mechanism that guarantees the migration job executes exactly once, in the right order, and cleans up after itself.

How Helm hooks address the problem

Helm provides built‑in hook annotations that let you attach a Kubernetes Job (or Pod) to a release lifecycle event. The most relevant hooks for database migrations are:

• pre-install – runs before any resources are created on the first install.
• pre-upgrade – runs before the new version of the chart is applied on an upgrade.

By annotating a Job with helm.sh/hook: pre-upgrade (or pre-install) you tell Helm to create and wait for that Job to reach a successful state before proceeding with the rest of the chart. The hook also respects the chart’s dependency order, so if your migration depends on a database StatefulSet defined in the same chart, that StatefulSet will be created first.

Hooks can be given a deletion policy. Using helm.sh/hook-delete-policy: hook-succeeded removes the Job resource after it finishes successfully, keeping the namespace tidy while preserving the Job’s logs for audit.

Worked example: a migration Job that updates a ConfigMap

Below is a minimal chart that demonstrates the pattern. The migration Job writes a timestamp into a ConfigMap; the main application simply reads that ConfigMap to prove the migration ran first.

Chart structure

mychart/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── migration-job.yaml
    ├── deployment.yaml
    └── configmap.yaml

templates/migration-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "mychart.fullname" . }}-migration
  annotations:
    "helm.sh/hook": pre-upgrade,pre-install
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "mychart.name" . }}
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrator
          image: {{ .Values.migration.image }}
          command: ["/bin/sh", "-c"]
          args:
            - |
              TIMESTAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
              kubectl create configmap {{ include "mychart.fullname" . }}-migration-status \
                --from-literal=last-run=$TIMESTAMP \
                --dry-run=client -o yaml | kubectl apply -f -

templates/configmap.yaml (observed by the app)

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "mychart.fullname" . }}-migration-status
data:
  last-run: ""

templates/deployment.yaml (main workload)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "mychart.name" . }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "mychart.name" . }}
    spec:
      containers:
        - name: app
          image: {{ .Values.app.image }}
          env:
            - name: MIGRATION_STATUS_CONFIGMAP
              value: {{ include "mychart.fullname" . }}-migration-status
          # The app reads the ConfigMap at startup to verify the migration ran.

Running the example

1. Package or chart directory is ready. Ensure you have kubectl configured for the target cluster and helm v3+ installed.
2. Install the release (first run):

   helm install myapp ./mychart --namespace demo --create-namespace

   Required permission: ability to create Jobs, ConfigMaps, and Deployments in the demo namespace.
3. Verify that the migration Job completed before the Deployment pods started:

   kubectl -n demo get jobs -l app.kubernetes.io/name=myapp
   kubectl -n demo get pods -l app.kubernetes.io/name=myapp
   kubectl -n demo get configmap myapp-migration-status -o yaml

   You should see the Job with status Completed and the ConfigMap’s last-run populated with a timestamp.
4. Upgrade the chart (no changes to values, just to trigger the hook):

   helm upgrade --install myapp ./mychart --namespace demo

   The hook runs again, updating the ConfigMap with a newer timestamp, while the existing Deployment pods are not restarted unless you changed the app image.
5. Check the ConfigMap again to confirm the timestamp changed:

kubectl -n demo get configmap myapp-migration-status -o yaml

If you want to inspect the Job logs for debugging, use:

kubectl -n demo logs job/myapp-migration

Trade‑offs and limitations

• Idempotency requirement: Because Helm may retry a hook if the Job fails or is pre‑empted, your migration script must be safe to run multiple times or detect that it has already been applied.
• Namespace scope: Hooks run in the same namespace as the release. If the migration needs cluster‑wide privileges (e.g., to modify a custom resource in another namespace), you must bind an appropriate ServiceAccount or RBAC role to the Job.
• Visibility: Once a hook succeeds and the deletion policy is hook-succeeded, the Job object is removed. Retaining logs for audit therefore requires either a different deletion policy (hook-failed or never) or exporting logs to a persistent store before the Job finishes.

Practical way to check the result: after each helm install or helm upgrade --install, run kubectl get job -l helm.sh/hook=pre-upgrade,pre-install to confirm the hook Job reached Completed status, and then inspect the ConfigMap or any other artifact your migration updates.

Actionable closing

Using Helm hooks for database migrations gives you a declarative, repeatable way to guarantee that schema changes run exactly once per release, in the correct order, and clean up after themselves. Start with a simple Job that writes a status marker (like the ConfigMap example above), make sure your migration script is idempotent, and apply the appropriate RBAC if elevated privileges are needed. This pattern reduces manual drift and lets you treat database upgrades as just another part of your Helm release lifecycle.