Using Argo CD ApplicationSet to Dynamically Generate Multi‑Cluster, Multi‑Environment Applications

Useful answer: let ApplicationSet create the Applications for you

When you need to deploy the same Helm chart to many clusters and environments (e.g., dev, staging, prod across dozens of clusters), writing an Application manifest for each combination is tedious and error‑prone. Argo CD’s ApplicationSet controller solves this by watching a generator (list, matrix, git, etc.) and automatically creating, updating, or deleting child Application resources that inherit a shared template. The result is a single source of truth for the deployment pattern, with per‑cluster or per‑environment variations injected via placeholders or Helm value files.

How it works: a matrix generator example

The following example shows an ApplicationSet that creates an Application for every combination of clusters listed in a Git‑stored clusters.yaml file and environments [dev, staging, prod]. The template uses the generated values to set the destination server, the path to the Helm chart, and the Application name.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-matrix
spec:
  generators:
    - list:
        elements:
          - cluster: dev-cluster
            server: https://dev-cluster.example.com:6443
          - cluster: prod-cluster
            server: https://prod-cluster.example.com:6443
    - matrix:
        generators:
          - list:
              elements:
                - env: dev
                - env: staging
                - env: prod
          - cluster: $list  # reference the first generator
  template:
    metadata:
      name: "guestbook-{{cluster.name}}-{{env}}"
      labels:
        team: platform
    spec:
      project: default
      source:
        repoURL: https://github.com/example/guestbook-chart.git
        targetRevision: HEAD
        path: charts/guestbook
        helm:
          valueFiles:
            - values.yaml                     # base values
            - values-{{env}}.yaml             # per‑env overlay (optional)
      destination:
        server: {{cluster.server}}
        namespace: guestbook
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

When the controller evaluates the list generator it yields two cluster objects. The nested matrix generator then pairs each cluster with each environment, producing six generator outputs. For each output the controller merges the generator fields (cluster.name, cluster.server, env) into the template, substitutes the placeholders, and creates (or updates) an Application with the resulting values.

What the controller does under the hood

1. Runs the generators according to their order and produces a set of key‑value pairs.
2. Applies a strategic merge patch: generator values override template defaults; labels and annotations are merged; Helm valueFiles lists are concatenated.
3. Creates an Application object, sets an ownerReference pointing to the ApplicationSet, and applies it via the Argo CD API.
4. On subsequent reconciliations (default every 180 s, configurable via requeueAfterSeconds) it compares the current generator output with the existing Application set and creates, updates, or deletes as needed.

Limits and common mistakes to watch for

Generator explosion

A matrix generator creates a Cartesian product. If you have 50 clusters and 10 environments you will get 500 Applications. Each Application consumes API server bandwidth and controller memory. Mitigation strategies:

• Set spec.generators[].matrix.maxClusterCount (available in Argo CD v2.9+) to limit how many clusters are processed per reconciliation.
• Split the matrix into multiple ApplicationSets (e.g., one per region) to keep each set under a few hundred Applications.
• Monitor the controller’s ApplicationSet pod metrics (cpu_usage_seconds_total, memory_usage_bytes) and increase resources if needed.

Destination server mismatch

The spec.destination.server value must exactly match a cluster that Argo CD knows about (added via argocd cluster add or a Cluster secret). If the server URL differs—even by a trailing slash or a different port—the generated Application stays in Pending with the error Destination server not found. Verify by running:

argocd cluster list

and ensure the server string in your generator matches one of the listed URLs.

Helm value merging nuances

ApplicationSet deep‑merges only map‑type values. Arrays (e.g., ingress.hosts) are replaced, not appended. If you need to add to an array per environment, use one of these approaches:

• Provide a separate values-{{env}}.yaml file via valueFiles and let the file contain the full array (overwrites the base).
• Keep the base array empty and define environment‑specific values in a Kustomize patch referenced through spec.source.plugin or a post‑render step.

Requeue interval and Git generator load

The default requeueAfterSeconds is 180 s. For a large mono‑repo scanned by a Git generator, each poll clones the repository, which can strain both the controller and the Git server. Reduce the clone depth:

generators:
  - git:
      repoURL: https://github.com/example/infra.git
      directories:
        - path: clusters/
      depth: 1          # shallow clone
      requeueAfterSeconds: 300  # increase if needed

Increasing the interval reduces load but slows detection of new directories; tune based on your repo size and acceptable latency.

Governance and deletion cascades

Starting with v2.8 you can enforce policies across all generated Applications using spec.governance. For example, require a team label:

spec:
  governance:
    policy:
      requireLabels:
        - team

If a template lacks the required label, the controller marks the generated Application as OutOfSync with a policy violation, preventing accidental drift.

By default, deleting an ApplicationSet cascades to delete all child Applications (cascade: true). If you want to preserve the live Kubernetes resources while removing Argo CD management, set:

spec:
  preserveResourcesOnDeletion: true

Verification checklist (do not claim execution)

• Apply the ApplicationSet and run argocd appset list -o wide to see the expected number of generated Applications.
• Add a new cluster to the generator list, wait for the next requeue, and confirm a new Application appears.
• Inspect a generated Application with argocd app get <name> -o yaml and verify that metadata.ownerReferences points to the ApplicationSet and that placeholders have been substituted.
• Test Helm value merging: define a base values.yaml and a values-{{env}}.yaml overlay, then run argocd app manifest <generated-app> and check that the rendered manifest contains the merged values.
• Remove a cluster from the generator list and ensure the corresponding Application is deleted (or preserved if preserveResourcesOnDeletion is set).
• Add a governance policy requiring a label, deliberately omit it from the template, and verify that the generated Application is reported as OutOfSync with a policy violation.

Following this pattern lets you treat clusters and environments as data rather than as repetitive boilerplate, while still retaining fine‑grained control over sync behavior, Helm overrides, and policy enforcement.