Diagnosing 403 Forbidden Errors Between Compute Engine and Cloud Storage

The Problem: 403 Forbidden Access to GCS

A Compute Engine (GCE) instance is unable to read or write files to a Google Cloud Storage (GCS) bucket, returning a 403 Forbidden error. This typically happens even when the VM is using the Compute Engine default service account, which many developers assume has implicit access to all project resources.

The takeaway: A 403 error in this context is rarely a network issue; it is almost always a mismatch between the IAM permissions granted to the service account and the API Access Scopes assigned to the VM instance.

Diagnostic Matrix: Identifying the Root Cause

Symptom	Likely Cause	Diagnostic Check
403 error on all buckets in the project	Restrictive VM API Access Scopes	Check metadata server scopes
403 error on one specific bucket only	Missing IAM Role for Service Account	Check Bucket IAM policy
403 error despite having 'Storage Admin' role	Legacy ACLs overriding IAM	Check Uniform Bucket-Level Access status

Step-by-Step Diagnostic Workflow

1. Verify the Active Identity

Before checking permissions, confirm which service account the VM is actually using. Run this command from the VM terminal:

gcloud auth list

Ensure the active account matches the service account you intend to authorize (e.g., [PROJECT_NUMBER]-[contact removed]).

2. Check API Access Scopes

Access Scopes are a legacy GCE feature that acts as a secondary filter on top of IAM. Even if a service account has "Owner" permissions, if the VM scope for Storage is set to "Disabled," the API call will fail with a 403.

Run this command on the VM to see the effective scopes:

curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes

Expected Result: You should see https://www.googleapis.com/auth/devstorage.read_only or https://www.googleapis.com/auth/cloud-platform. If neither is present, the VM is blocked at the scope level.

3. Validate IAM Permissions

If scopes are correct, check the IAM policy. The service account needs a role that permits the specific action (e.g., storage.objects.get for downloading).

• Storage Object Viewer (roles/storage.objectViewer): Allows listing and reading objects.
• Storage Object Creator (roles/storage.objectCreator): Allows uploading objects.
• Storage Object Admin (roles/storage.objectAdmin): Full control over objects.

Applying the Fixes

Fix A: Updating API Access Scopes

Risk: You cannot change scopes while a VM is running. The instance must be stopped.

1. Stop the VM instance via the Console or gcloud compute instances stop [INSTANCE_NAME].
2. Edit the instance. Under Cloud API access scopes, select Allow full access to all Cloud APIs.
3. Start the VM.

Note: Setting "Allow full access" is the modern best practice; it shifts the security burden entirely to IAM, which is more granular and easier to audit.

Fix B: Granting IAM Roles

If scopes are open but access is denied, grant the role at the bucket level to maintain the principle of least privilege.

# Run this from a workstation with Project Editor/Owner permissions
gcloud storage buckets add-iam-policy-binding gs://[BUCKET_NAME] 
    --member="serviceAccount:[PROJECT_NUMBER]-[contact removed]" 
    --role="roles/storage.objectViewer"

Fix C: Enabling Uniform Bucket-Level Access

If you see inconsistent behavior (some files accessible, others not), the bucket may be using legacy Access Control Lists (ACLs). Enable Uniform Bucket-Level Access to ensure IAM is the sole authority.

gcloud storage buckets update gs://[BUCKET_NAME] --uniform-bucket-level-access

Warning: This disables all fine-grained ACLs. Ensure all required users have the appropriate IAM roles before enabling.

Verification and Escalation

To verify the fix, attempt to list the bucket contents directly from the VM:

gcloud storage ls gs://[BUCKET_NAME]

If the command returns the list of files without a 403 error, the issue is resolved.

When to Escalate

If you have verified that 1) the VM has cloud-platform scopes, 2) the service account has Storage Object Admin, and 3) Uniform Bucket-Level Access is enabled, but you still receive a 403, check for VPC Service Controls (VPC-SC). VPC-SC creates a security perimeter that blocks API requests based on network origin, regardless of IAM permissions.