Using Argo CD ApplicationSet for Minimal Multi‑Environment Deployments

Requirements

\n

To run the design described here you need:

\n

\n
• A Git repository that holds Helm charts or Kustomize overlays, with one sub‑directory per target environment (e.g., dev/, staging/, prod/).
\n
• An Argo CD installation where the application-controller pod is running and has network access to the Git host.
\n
• A service account token for the controller that is scoped to read‑only access on the repository.
\n
• A target Kubernetes cluster where Argo CD will create namespaced resources (the cluster can be the same that runs Argo CD or a separate one).
\n

\n\n

Smallest Suitable Design

\n

The minimal construct that satisfies the requirements is a single ApplicationSet resource that uses a List generator to enumerate environment directories and a template Application that:

\n

\n
1. Points to the Helm chart (or Kustomize overlay) inside the matched directory.
\n
2. Sets syncWave annotations to enforce an ordered creation of resources (e.g., namespaces first, then workloads).
\n
3. Enables the built‑in health assessment so Argo CD can mark the application Healthy only when all resources report Ready conditions.
\n

\n

Below is an example manifest. Replace placeholders surrounded by <> with your actual values.

\n

apiVersion: argoproj.io/v1alpha1\nkind: ApplicationSet\nmetadata:\n  name: env-appset\n  namespace: argocd\nspec:\n  generators:\n    - list:\n        elements:\n          - env: dev\n          - env: staging\n          - env: prod\n  template:\n    metadata:\n      name: '{{env}}-app'\n    spec:\n      project: default\n      source:\n        repoURL: <GIT_REPO_URL>\n        targetRevision: HEAD\n        path: '{{env}}'   # points to env-specific folder\n        helm:\n          valueFiles:\n            - values.yaml\n      destination:\n        server: <TARGET_CLUSTER_API_SERVER>\n        namespace: '{{env}}'\n      syncPolicy:\n        automated:\n          prune: true\n          selfHeal: true\n        syncOptions:\n          - CreateNamespace=true\n      # Optional: enforce creation order with syncWave\n      # (add annotations to resources in your chart, e.g., \"argocd.argoproj.io/sync-wave\": \"0\")\n

\n

Apply the manifest with:

\n

# Run in a context where kubectl can reach the Argo CD cluster\nkubectl apply -f env-appset.yaml\n

\n

The controller will:

\n

\n
• Read the Git repository using its read‑only token.
\n
• Create three Application objects: dev-app, staging-app, prod-app.
\n
• Attempt to sync each application to its respective namespace.
\n

\n\n

Trust and Data Boundaries

\n

The application-controller runs under a dedicated service account (often argocd-application-controller). Its trust boundaries are:

\n

\n
• Git side: The controller only performs git clone / git fetch operations using the supplied read‑only token. It cannot push, create branches, or modify repository contents.
\n
• Cluster side: The controller creates namespaced resources only inside the namespaces specified in the destination field of each generated Application. It does not acquire cluster‑wide privileges unless you explicitly grant them via RBAC.
\n
• Data exposure: Secrets are only visible to the controller if the template explicitly references them (e.g., via secretRef in a Helm values file). Keeping the token read‑only and limiting repository contents to non‑secret configuration reduces leakage risk.
\n

\n

To reinforce these boundaries, consider:

\n

\n
• Creating a dedicated GitHub/GitLab read‑only deploy key** or token with the repo:read scope only.
\n
• Applying a network policy that allows the controller pod to egress only to the Git host’s IP on port 443.
\n
• Using Argo CD’s RBAC to restrict which projects or namespaces the controller may manage.
\n

\n\n

Operational Checks

\n

After applying the ApplicationSet, verify the following:

\n

\n
1. Application objects appear:
\n

\n

kubectl -n argocd get applicationset env-appset -o yaml\nkubectl -n argocd get applications -l argocd.argoproj.io/application-set=env-appset\n

\n

You should see three Application resources, each with a status.sync of Synced and status.health of Healthy (assuming the chart deploys successfully).

\n

\n
1. Environment‑specific sync: Change a value in dev/values.yaml and push.
\n

\n

# Example: edit dev/values.yaml, commit, push\nkubectl -n argocd get application dev-app -w\n

\n

Only the dev-app should transition to OutOfSync and then back to Synced; the other two applications remain unchanged.

\n

\n
1. Health assessment: Introduce a failing resource (e.g., set a replica count to zero) in one environment and verify that the corresponding Application shows Health: Degraded and Sync: OutOfSync.
\n

\n\n

Failure Modes

\n

Understand how the design behaves when things go wrong:

\n

\n
• Git connectivity loss: The controller detects the error, logs a warning, and enters exponential backoff (starting at ~10 s, doubling each attempt). Applications managed by the ApplicationSet become Degraded. When connectivity returns, the controller resumes polling and reconciles any missed commits.
\n
• Malformed generator: If the List generator references a non‑existent directory, the controller creates an Application that fails to sync because the path cannot be found. The Application stays OutOfSync with a Missing health status; logs show \"failed to resolve path\".
\n
• Controller resource exhaustion: With hundreds of environments, the controller’s CPU and memory usage rise linearly. Monitor the metrics argocd_application_set_controller_operations_total and argocd_application_set_controller_memory_usage_bytes. If usage approaches limits, consider sharding (multiple ApplicationSet objects each handling a subset) or switching to a Cluster generator with pagination.
\n

\n\n

When to Change the Design

\n

The minimal single‑ApplicationSet approach is appropriate when:

\n

\n
• The number of environments is modest (typically < 50) and each environment uses the same chart structure.
\n
• You want a single source of truth for the set of applications and prefer the controller to handle creation/deletion automatically as directories are added or removed.
\n
• Read‑only Git access satisfies your security policy.
\n

\n

Consider moving to a more complex design if:

\n

\n
• You need different charts or Helm values per environment that cannot be expressed via simple directory overrides.
\n
• You require fine‑grained sync policies (e.g., some environments manual, others automated) that vary per‑application.
\n
• Your scale reaches hundreds of environments and the controller’s resource consumption becomes a bottleneck; in that case, split the ApplicationSet by function or region, or use the Cluster generator with a paginated list of clusters.
\n
• You must enforce stricter network isolation, such as preventing the controller from reaching any external Git host; then you would mirror the repository inside the cluster and use an Git generator pointing to the internal mirror.
\n

\n\n

Practical Verification Checklist

\n

\n
1. Deploy Argo CD (version 2.9+ recommended) in a test cluster.
\n
2. Create a read‑only Git token and store it as a Kubernetes secret (argocd-git-token) referenced by the application-controller via the ARGOCD_GIT_TOKEN env var or the secret field in the argocd-secret.
\n
3. Apply the ApplicationSet manifest shown above.
\n
4. Confirm three Application objects are present and healthy.
\n
5. Make a change in one environment’s directory and observe that only the corresponding application reconciles.
\n
6. Simulate a Git outage (e.g., block outbound TCP 443 to the Git host with a network policy or iptables) and verify the controller logs a backoff warning and the applications become degraded.
\n
7. Restore connectivity and confirm the applications return to Synced and Healthy.
\n

\n

By following these steps you can validate that the minimal ApplicationSet design meets the requirements, respects trust boundaries, and provides observable operational feedback.