Automating Multi‑Cluster Deployments with Argo CD ApplicationSets and the Git Generator

Desired Outcome

Automatically create, update, and delete Argo CD Application resources for every cluster defined in a Git repository, without manual UI or CLI intervention.

Prerequisites

• Argo CD v2.7+ deployed in a Kubernetes cluster (namespace argocd).
• ApplicationSet controller installed (see official docs).
• Git repository containing a /clusters directory with one YAML file per target cluster (e.g., us-east-1.yaml, eu-west-2.yaml).
• RBAC granting the ApplicationSet controller permission to create, update, and delete argoproj.io/v1alpha1/Application resources in the argocd namespace.
• Network connectivity from the controller to the Git provider and to each target cluster’s API server.

Setup Procedure

1. Prepare the Git source
   • Clone or create a repo at https://github.com/example/org-configs.git.
   • Inside the repo, add a /clusters directory.
   • Place a YAML file per cluster. Example us-east-1.yaml:

     clusterName: us-east-1
     apiServer: https://api.us-east-1.eks.amazonaws.com
     namespace: prod
     

2. Define the ApplicationSet
   • Create a file appsets/multi-cluster.yaml with the following content:

     apiVersion: argoproj.io/v1alpha1
     kind: ApplicationSet
     metadata:
       name: multi-cluster-apps
       namespace: argocd
     spec:
       generators:
       - git:
           repoURL: https://github.com/example/org-configs.git
           revision: main
           directories:
           - path: clusters
       template:
         metadata:
           name: "{{clusterName}}-app"
           namespace: argocd
         spec:
           project: default
           source:
             repoURL: https://github.com/example/app-repo.git
             targetRevision: HEAD
             path: "{{namespace}}"
           destination:
             server: "{{apiServer}}"
             namespace: "{{namespace}}"
           syncPolicy:
             automated:
               prune: true
               selfHeal: true
     

   • Notice the Go template placeholders ({{clusterName}}, {{apiServer}}, {{namespace}}) that map Git file fields to the Application spec.
3. Apply the ApplicationSet
   • Run on the cluster where Argo CD is installed:

     kubectl apply -f appsets/multi-cluster.yaml
     

   • Permissions: the command must be executed as a user with apply rights in the argocd namespace.
   • Expected output: applicationset.argoproj.io/multi-cluster-apps created.

Verifying Deployment

• List the child Applications:

  kubectl get applications -n argocd
  

• You should see entries like us-east-1-app and eu-west-2-app.
• Check sync status in Argo CD UI or via CLI:

  argocd app list -n argocd
  

• For a specific app, view logs:

  argocd app get us-east-1-app -n argocd
  

• Confirm that the destination server matches the apiServer field from the Git file.

Adding a New Cluster

1. Create clusters/asia-south1.yaml with appropriate values.
2. Push the change to main.
3. The ApplicationSet controller polls the repo (default every 60 s). Within a minute you should see a new Application asia-south1-app appear.
4. Verify with kubectl get applications -n argocd | grep asia-south1.

Removing a Cluster

1. Delete the YAML file from the /clusters directory.
2. Push the change.
3. The controller automatically prunes the corresponding Application due to the syncPolicy.automated.prune: true setting.
4. Check that the Application is gone: kubectl get applications -n argocd | grep asia-south1 should return nothing.

Common Pitfalls & Recovery

• Cascading Deletions
  • If the Git generator pattern is mis‑configured (e.g., scanning a directory that includes unrelated files), the controller may interpret all files as cluster definitions and delete every managed Application.
  • Recovery: Inspect argocd/applicationset logs for the offending ApplicationSet. Temporarily patch the spec to remove the directories block or set prune: false. Then re‑apply a correct spec.
• API Rate Limiting
  • High Git polling frequency can trigger Git provider limits.
  • Solution: Reduce syncInterval in the ApplicationSet spec or use git.pollIntervalSeconds to increase the interval.
• RBAC Issues
  • Missing permissions will cause the controller to fail creating Applications.
  • Check the controller logs for “forbidden” errors. Ensure the ServiceAccount used by the controller has a ClusterRole binding that allows create, update, and delete on argoproj.io/v1alpha1/Application.

Limitations

• ApplicationSets cannot currently manage Applications across namespaces that are not owned by the same Argo CD instance. Deploying to clusters that require separate Argo CD instances requires a separate ApplicationSet per instance.
• Templates are limited to Go templating; complex logic may need external preprocessing.
• Git polling is the only supported source for the Git generator; other generators (Helm, Cluster, List) are not covered here.

Summary

Using Argo CD ApplicationSets with the Git Generator lets you treat a Git repo as a single source of truth for multi‑cluster deployments. Add or remove a YAML file and the system automatically creates or prunes the corresponding Applications. By following the steps above, you can set up this pattern, verify that it works, and recover from common configuration mistakes.