Managing Multi‑Cluster Fleets with Argo CD ApplicationSet Matrix Generator

The problem: scaling GitOps across dozens of clusters

Teams that adopt Argo CD for GitOps quickly hit a wall when the same application must run in multiple clusters, each with several environments (dev, staging, prod) and possibly regional variations. Maintaining a separate Application manifest for every combination becomes brittle — copy‑paste errors drift, and any change to the Helm chart or sync policy requires editing dozens of files.

Why ApplicationSet’s Matrix generator fits

The ApplicationSet controller (installed as argocd-applicationset-controller since Argo CD v2.3, GA in v2.5) watches a single ApplicationSet CRD and materialises child Application resources from a template. The Matrix generator builds the Cartesian product of two or more parameter lists, so you declare the clusters once, the environments once, and the controller creates every needed Application automatically.

Worked example: a Helm‑based microservice across three clusters and two environments

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payment-service
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  environment: production
          - git:
              repoURL: https://github.com/example/helm-charts.git
              revision: main
              directories:
                - path: charts/payment-service/*/values.yaml
  template:
    metadata:
      name: '{{cluster.name}}-{{path.basename}}'
      namespace: '{{cluster.metadata.labels.environment}}'
    spec:
      project: '{{cluster.metadata.labels.team}}'
      source:
        repoURL: https://github.com/example/helm-charts.git
        targetRevision: main
        path: 'charts/payment-service/{{path.basename}}'
        helm:
          valueFiles:
            - 'values.yaml'
            - '{{cluster.name}}/values-{{path.basename}}.yaml'
      destination:
        server: '{{cluster.server}}'
        namespace: '{{cluster.metadata.labels.environment}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

In this manifest:

• The clusters generator selects all Argo CD cluster secrets labelled environment: production.
• The git generator discovers each sub‑directory under charts/payment-service/ that contains a values.yaml (each directory represents a service variant).
• The matrix crosses the two lists, yielding N clusters × M variants child Applications.
• Go templating (Sprig functions) injects cluster‑specific values files and destination namespaces.

Dry‑run before committing

Run the following from a workstation with argocd CLI configured (requires argocd admin or a role that can create ApplicationSets in the target namespace):

argocd appset generate payment-service.yaml --dry-run

The command prints every rendered Application without persisting them. Verify that the name, destination.server, and helm.valueFiles fields match expectations.

Inspect the live fleet

After applying the manifest (kubectl apply -f payment-service.yaml), use these checks (run in the management cluster where Argo CD lives):

# List ApplicationSets and their generator status
kubectl get applicationset -A -o wide

# Show child Applications created by this set
kubectl get applications -l argocd.argoproj.io/instance=payment-service -o custom-columns=NAME:.metadata.name,CLUSTER:.spec.destination.server,NAMESPACE:.spec.destination.namespace,SYNC:.status.sync.status,HEALTH:.status.health.status

The label argocd.argoproj.io/instance is automatically added by the controller and equals the ApplicationSet name.

Trade‑offs and limitations

• Combinatorial explosion: 10 clusters × 5 environments × 3 variants = 150 Applications. Each child consumes controller memory and UI rows. Validate the product size before applying (e.g., argocd appset generate ... --dry-run | grep -c '^apiVersion:').
• Git generator memory pressure: Large repos with thousands of directories can OOM the controller. Mitigate with requeueAfterSeconds and restrictive directories globs.
• RBAC inheritance: The controller impersonates the ApplicationSet creator. If a non‑admin user creates the set in a Project that only allows cluster cluster-a, any matrix entry referencing cluster-b will be rejected at child‑Application creation time.
• Template errors surface only in status: A missing key or bad Sprig call yields a TemplateError condition on the ApplicationSet, not a Kubernetes validation error. Add a CI step that runs argocd appset generate and fails on non‑zero exit.

Actionable next steps

1. Inventory the clusters you manage and label their Argo CD secrets consistently (e.g., environment, team, region).
2. Structure your Helm chart repository so each service variant lives in its own directory with a base values.yaml and optional cluster‑specific overrides.
3. Write a single ApplicationSet using the Matrix generator as shown; commit it to your GitOps repo.
4. Add a pipeline job that runs argocd appset generate <file> --dry-run on every PR; fail the build if the command returns an error or the generated count exceeds a defined threshold.
5. Monitor the ApplicationSet status (kubectl get applicationset payment-service -o yaml) for TemplateError or GeneratorError conditions.
6. When decommissioning a cluster, remove its secret or label; the controller will delete the corresponding child Applications within the default 180 s requeue interval.

By centralising the fleet definition in one ApplicationSet, you gain a single source of truth, automatic cleanup, and a clear view of sync/health across every cluster‑environment pair — without maintaining hundreds of near‑identical Application manifests.