Diagnosing Gardener Shoot Provisioning Failures Due to Missing or Invalid Provider Credentials

Problem Statement

A Gardener Shoot that ends in ProvisioningFailed with an error mentioning “provider credentials not found” or “invalid credentials” indicates that Gardener cannot authenticate to the underlying cloud provider. This guide walks through a repeatable diagnostic process, linking each observation to a specific fix.

Recognizable Condition

When the Shoot’s Status shows Phase: ProvisioningFailed and the Message contains words like “credentials”, “provider”, or “auth”, the failure is almost certainly due to Gardener being unable to read or use the Secret that holds the cloud provider’s access keys.

Key Symptoms

• Shoot status ProvisioningFailed with message “provider credentials not found” or “credential validation failed”.
• Gardener controller logs contain entries such as “failed to fetch provider credentials” or “credential validation failed”.
• Cloud provider API returns HTTP 401 or 403 when Gardener attempts to create resources.

Cause & Diagnostic Table

Possible Cause	What to Look For	First Check
Secret referenced by Shoot is missing or empty	Gardener log: “secret … not found”	Run kubectl get secret <secret-name> -n <seed-namespace>
Secret data keys are incorrect or incomplete	Missing required key (e.g., aws_access_key_id)	Decode a key: kubectl get secret <secret-name> -n <seed-namespace> -o jsonpath='{.data.aws_access_key_id}' | base64 -d
Provider config in Shoot spec points to a non‑existent Secret	Gardener log: “provider config secret … does not exist”	Inspect spec.providerConfig.secretName in the Shoot YAML
Cloud provider rejects credentials (API 401/403)	Gardener log: “credential validation failed”	From a pod in the seed cluster, curl the provider’s API endpoint
Network path to provider API is blocked	Gardener log: “failed to connect to provider endpoint”	Ping or curl from a seed pod to the API URL

Ordered Checks

1. Confirm Shoot status
   Run

   kubectl get shoot <shoot-name> -n <namespace> -o yaml | grep -i provisioningfailed -A5

   Look for Phase: ProvisioningFailed and the Message field. No special permissions beyond read access to the Shoot namespace are required.
2. Identify the Secret used by the Shoot
   Execute

   kubectl get shoot <shoot-name> -n <namespace> -o yaml | grep -A2 providerConfig

   Note the secretName and its namespace (usually the seed namespace). This step requires read access to the Shoot object.
3. Verify Secret existence and content
   Run

   kubectl get secret <secret-name> -n <seed-namespace> -o yaml

   If the command returns NotFound, the Secret is missing. If it exists, examine the data section.
4. Decode required keys
   For AWS, check both aws_access_key_id and aws_secret_access_key:
   

   kubectl get secret <secret-name> -n <seed-namespace> -o jsonpath='{.data.aws_access_key_id}' | base64 -d

   Repeat for aws_secret_access_key. For GCP, check google_application_credentials; for Azure, check azure_client_id, azure_client_secret, azure_tenant_id. A blank output or decoding error indicates a missing or malformed key.
5. Examine Gardener controller logs for credential messages
   Run

   kubectl logs -n gardener-system deployment/gardener-controller-manager | grep -i credential

   Look for lines that mention “failed to fetch”, “validation failed”, or “secret not found”. Access to the gardener-system namespace is needed.
6. Test API connectivity from the seed cluster
   Create a temporary pod in the seed namespace and curl the provider’s endpoint:
   

   kubectl run -i --tty test --image=alpine --restart=Never --namespace=<seed-namespace> --command -- sh -c 'apk add --no-cache curl && curl -I https://api.aws.amazon.com/'

   Replace the URL with the appropriate endpoint for GCP (https://www.googleapis.com/) or Azure (https://management.azure.com/). A successful HTTP 200 indicates network reachability; a timeout or connection refused suggests a network block. The pod runs with default service account; ensure it has permission to execute exec or run in the namespace.

Fixes Tied to Findings

Missing or Empty Secret

Re‑create the Secret with valid credentials obtained from the cloud provider’s console or CLI. Example for AWS:

kubectl create secret generic aws-cred-<shoot-name> --namespace=<seed-namespace> --from-literal=aws_access_key_id=<ACCESS_KEY_ID> --from-literal=aws_secret_access_key=<SECRET_ACCESS_KEY>

After creation, Gardener will retry the Shoot automatically; you can also kubectl apply -f <shoot-yaml> to trigger an immediate reconciliation.

Incorrect or Missing Data Keys

Instead of patching, recreate the Secret with the complete set of keys. For AWS:

kubectl create secret generic aws-cred-<shoot-name> --namespace=<seed-namespace> --from-literal=aws_access_key_id=<ACCESS_KEY_ID> --from-literal=aws_secret_access_key=<SECRET_ACCESS_KEY> --dry-run=client -o yaml | kubectl apply -f -

This approach avoids exposing keys in shell history.

Provider Config Refers to Non‑existent Secret

Edit the Shoot to point to the correct Secret:

kubectl edit shoot <shoot-name> -n <namespace>

In the editor, change spec.providerConfig.secretName to match an existing Secret. Save and exit; Gardener will notice the change and retry.

Cloud Provider Rejects Credentials (API 401/403)

Verify that the IAM policy or service account attached to the credentials grants the permissions Gardener needs. For a typical AWS Shoot, the access key should allow ec2:*, iam:*, and s3:* actions. Attach the appropriate policy via the AWS console or CLI, then update the Secret with the new key pair if necessary.

Network Path Blocked

Check the seed cluster’s NetworkPolicy, egress rules, and any cloud‑level security groups. To test, temporarily allow all egress:

kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-test
  namespace: <seed-namespace>
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - {}
EOF

If the connectivity test now succeeds, tighten the policy to only the required API endpoints. Remember to delete the test policy after verification.

Escalation Criteria

• If the Secret exists, contains all required keys, and network tests succeed, yet Gardener still logs credential errors, open an issue with the Gardener maintainers and include the relevant controller logs and Shoot YAML.
• When using a supported Gardener version (≥ v1.12) and the provider‑specific credential format has changed, consult the provider’s documentation in the Gardener user guide.
• For environments where many Shoots share a Secret, coordinate credential rotation to avoid simultaneous failures; update all referencing Secrets before the old credentials expire.

Limitations & Practical Checks

Gardener’s credential validation logic varies slightly between cloud providers; always refer to the provider‑specific section of the Gardener documentation for exact key names and required IAM scopes.

After applying a fix, monitor the Shoot’s phase:

watch kubectl get shoot <shoot-name> -n <namespace> -o jsonpath='{.status.phase}'

A transition from ProvisioningFailed to Provisioning and then to Running indicates the issue is resolved. If the Shoot remains in ProvisioningFailed, repeat the ordered checks, paying attention to any new log messages.