Argo CD ApplicationSet: Designing a Fleet‑wide Deployment Pattern with Trust Boundaries

Problem and Takeaway

Many teams need to keep a single deployment pattern consistent across multiple Kubernetes clusters or environments, while ensuring that every change is reviewable in Git and that no Application is hand‑created. The core issue is balancing Git‑driven desired state with secure, automated reconciliation, and preventing a bad change from propagating to every generated Application.

Takeaway: Use Argo CD’s ApplicationSet with a Git generator to create one Application per target, store manifests and generator inputs in a protected Git repository, and enforce separate trust boundaries for Git authors and Argo CD execution credentials.

Smallest Suitable Design

Git Repository Layout

• manifests/: Kubernetes YAML files (e.g., deployment.yaml, service.yaml).
• generator/: Input files for the generator (e.g., cluster-config.yaml, image-tag.yaml).
• applicationset.yaml: The ApplicationSet manifest that references the generator and target clusters.

All files reside in a single Git repository that Argo CD can read. The repository is the source of truth; any change must go through a PR and be merged before it reaches production.

ApplicationSet Manifest (example)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: fleet‑apps
spec:
  generators:
  - git:
      repoURL: 
      branch: main
      directories:
      - path: manifests/
      - path: generator/
  base:
  - repoURL: 
    path: applicationset.yaml
  scope: Namespaced
  project: default
  targets:
  - cluster:
      name: cluster‑prod‑us-east
      namespace: production
  - cluster:
      name: cluster‑staging‑eu-west
      namespace: staging

The targets section defines the clusters where each generated Application will be created. Each cluster gets its own Application, ensuring isolated reconciliation.

Trust and Data Boundaries

Git Author Trust

Git authors can modify manifests and generator inputs. Since these files drive the desired state, they represent the primary trust boundary for *what* is deployed.

Argo CD Execution Trust

Argo CD uses repository credentials (often a service account token) to clone the repo and cluster credentials (e.g., a Kubeconfig) to apply manifests. These credentials supply the *authority* to write to the cluster. Protect both paths:

• Restrict write access to the Git repository (e.g., via branch protection, CODEOWNERS).
• Limit which Argo CD project and service account may use the cluster secret.

Separate IAM roles for Git write and Argo CD execution prevent a compromised author from directly controlling cluster state.

Operational Checks

Inspect the Generated Application Inventory

After a change, verify that the correct number of Applications exists and that each points to the intended cluster and namespace:

1. Run argocd app list --output yaml to view all Applications.
2. Check the spec.generator and status.conditions fields for each Application.

If an Application is missing, the generator may have failed to pick up the change; investigate the Git directory and the ApplicationSet’s generator configuration.

Sync and Health Status

Sync status indicates whether the desired state in Git matches the cluster state. Health checks verify that the deployed resources are healthy:

1. Use argocd app get to see sync.status (Synced, OutOfSync, Degraded).
2. Run argocd app compare to view a diff between Git and the cluster.

Do not assume UI freshness equals correctness; always verify the actual sync and health conditions.

Drift and Comparison Errors

If a manual change occurs outside Argo CD, the Application will show a drift condition. Use argocd app diff to detect unexpected changes.

Failure Modes and Containment

• Stale Git state: The repository may not reflect the latest commit due to a delayed pull or a broken webhook. Verify that the ApplicationSet’s git generator is polling correctly.
• Expired repository credentials: If the token used by Argo CD to access the repo expires, the generator cannot fetch manifests. Monitor secret rotation and set up alerts for credential expiration.
• Delayed updates: The generator may batch changes; a time‑sensitive rollout could appear later than expected. Use short polling intervals or webhook triggers for urgent updates.
• Invalid templates: A malformed generator input (e.g., missing required fields) can cause the Application to be created with an invalid spec, leading to sync failures. Validate generator inputs with schema checks or CI pipelines.
• Partial multi‑cluster rollout: If a cluster becomes unreachable, only the Applications targeting that cluster will stay OutOfSync. Ensure each cluster has its own credentials and network path.
• Compromised author: A malicious Git user could add a new target that points to an unauthorized cluster. Enforce project‑level RBAC and limit which identities can modify the ApplicationSet.

When to Change Direction

If your workflow requires explicit approvals, staged promotion, disconnected clusters, or stronger tenant isolation, the simple ApplicationSet model may be insufficient. Consider:

• Adding a manual argocd app sync step after a PR merge.
• Using Argo Rollouts or separate Argo CD projects per environment to enforce staged delivery.
• Introducing a dedicated “promotion” ApplicationSet that only creates Applications in a target cluster after a gatekeeper check.
• Employing a side‑car controller that watches a separate Git repo for promotion signals, then updates the target ApplicationSet.

Practical Verification Workflow

1. Pre‑flight checks: Confirm the ApplicationSet CRD and controller are installed (kubectl get crd applicationsets.argoproj.io). Verify that the Argo CD service account has read access to the Git repo secret and cluster secret.
2. Pilot deployment: Apply the ApplicationSet to a non‑production cluster: kubectl apply -f applicationset.yaml. Then create a small change in manifests/ (e.g., update a label) and commit. Observe the generated Application appear, sync, and become healthy.
3. Failure injection: a) Edit the Git repo URL to an invalid location and ensure the Application shows a credential error. b) Revoke the repo token and confirm the Application remains OutOfSync until credentials are refreshed. c) Break the network to a target cluster and verify that only its Applications stay OutOfSync.
4. Review permissions: Use argocd project get to see which identities can act on the project, and audit Argo CD audit logs for any unexpected template changes.

Conclusion

Argo CD’s ApplicationSet provides a lightweight, Git‑centric way to manage a fleet of Kubernetes Applications while keeping the desired state fully reviewable. By clearly separating Git author trust from Argo CD execution trust, and by performing regular inventory, sync, health, and drift checks, teams can achieve consistent deployments across clusters. When requirements shift toward stricter governance—approvals, staged rollouts, or isolated tenants—augment the basic model with explicit workflow steps or alternative reconciliation topologies. Always verify the installed components, run a pilot, and test failure scenarios before scaling the pattern to production.