Using Rancher Fleet for GitOps‑driven Multi‑Cluster Deployments

The problem: keeping dozens of downstream clusters in sync

When you manage many Kubernetes clusters through Rancher, applying the same Helm chart or Kustomize overlay to each one manually becomes error‑prone and slow. A missed update can leave a cluster vulnerable, while a rushed rollout can cause widespread instability. You need a declarative, Git‑based way to store the desired state and have the system reconcile drift automatically.

Thesis: Rancher Fleet provides a lightweight controller that watches a Git repo and applies resources to selected clusters, giving you a single source of truth with progressive rollout capabilities.

How Fleet works (brief)

The Fleet controller runs inside the Rancher management server. It watches a Git repository for changes to a FleetWorkspace object, which bundles a Helm chart (or raw manifests) together with a values.yaml or Kustomize overlay. When the workspace is updated, the controller creates or updates a FleetBundle for each target cluster, and the Rancher API pushes the resources to the cluster’s API server. The process is declarative: the Git repo is the source of truth, and the controller continuously reconciles any drift.

Worked example: deploying NGINX Ingress to three downstream clusters

1. Prepare the Git repo (run on your workstation, you need push access):

   mkdir fleet-nginx && cd fleet-nginx
   helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
   helm pull ingress-nginx/ingress-nginx --untar --version 4.9.0
   mv ingress-nginx/ charts/
   # Add a simple values.yaml
   cat > charts/values.yaml <<'EOF'
   replicaCount: 2
   EOF
   # Initialize a Fleet workspace
   fleetctl create workspace nginx-workspace --bundle charts/ --cluster-selector 'environment=production'
   git init
   git add .
   git commit -m "Add NGINX Ingress bundle"
   git remote add origin [contact removed]:example/fleet-nginx.git
   git push -u origin master
   

2. Where to run: The fleetctl commands require a kubeconfig that points to the Rancher management server with the admin or fleet-admin role. The Git push needs write permission to the repository.
3. What happens: The Fleet controller detects the new commit, creates a FleetWorkspace named nginx-workspace in the fleet-default namespace, and for each cluster matching the selector it creates a FleetBundle. Within seconds to minutes (depending on network latency) the controller applies the Helm release to each downstream cluster.
4. Expected checks: After a few minutes, verify on the management server:

   kubectl get fleetworkspace -n fleet-default nginx-workspace -o yaml
   

   Look for status: applied under status.conditions. Then, on any downstream cluster (you need its kubeconfig):

   helm list -n ingress-nginx
   

   You should see a release named something like nginx-workspace-ingress-nginx with replicaCount 2.

Trade‑offs and limitations

• Network dependency: The controller must be able to reach the API server of every downstream cluster. Intermittent connectivity can delay reconciliation, leaving clusters temporarily out of sync.
• Resource usage on the management server: Each watched workspace and each target cluster adds to the controller’s memory and CPU footprint. Large fleets (hundreds of clusters) may require scaling the Rancher server or allocating extra resources to the fleet-controller deployment.
• Progressive rollout granularity: Fleet supports batch sizes and canary annotations, but the smallest unit is a cluster. You cannot roll out a change to a subset of namespaces within a single cluster without additional tooling.

Verification and rollback

To confirm Fleet is healthy, check the controller logs:

kubectl logs -n fleet-default deploy/fleet-controller

Look for lines like Synced workspace nginx-workspace or error messages indicating Get https://<cluster-api>: dial i/o timeout. If a bundle fails to apply, the workspace status will show Failed and the log will contain the underlying Helm or API error.

Rollback is straightforward because the desired state lives in Git. Reverting the commit that introduced the faulty bundle (or correcting the values.yaml) triggers a new reconciliation, and Fleet will bring the clusters back to the previous state. No separate rollback command is needed; the Git revert is the rollback mechanism.

Actionable closing

Start small: create a test repository with a simple Helm chart, commit it, and watch the FleetWorkspace transition to applied. Monitor the controller logs and verify the release on a single downstream cluster before adding more clusters to the selector. Once you’re comfortable with the workflow, enable batch sizes or canary annotations to promote changes safely across your fleet.