Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Node.js, PHP and APIs

Google Secret Manager says access denied although the roles look right

ErrorPERMISSION_DENIED when accessing a secret version (permission secretmanager.versions.access)

Short answer

Secret Manager checks one thing: does the identity that makes the call hold secretmanager.versions.access on that secret? When the role looks right, the call usually runs as a different identity, the role is on another secret or project, the role lacks the permission, a VM scope blocks it, or the grant is still propagating.

  • Applies to Secret Manager API, Cloud Run, Compute Engine, gcloud CLI, Node.js 22
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Show who has access to the secret you read. Then compare that list with the identity your code really runs as, which is often not the one you granted.

    gcloud secrets get-iam-policy SECRET_ID
  2. 2Match what you see, then jump to its fix

    1. 1The policy lists a service account or user with Secret Accessor, but the call still fails.The code runs as a different identity than the one you grantedFix
    2. 2The principal holds Editor, Viewer or a role on the project, but the policy of this secret does not list Secret Accessor.The role does not include the permission, or sits on another secret or projectFix
    3. 3The workload runs on a Compute Engine VM, the right service account holds the role, and the call is still denied.The VM or GKE node does not have the cloud-platform access scopeFix
    4. 4The policy is correct, and the same call succeeds a few minutes later without any change.The grant was made a moment ago and has not propagatedFix

What does the error look like?

  • PERMISSION_DENIED when accessing a secret version (permission secretmanager.versions.access)
  • The request failed because the instance could not start successfully.

Cause 1 of 4

The code runs as a different identity than the one you granted

1Check

Application Default Credentials pick a credential in a fixed order: the file named by GOOGLE_APPLICATION_CREDENTIALS, then the file made by gcloud auth application-default login, then the attached service account. The ADC credentials are separate from the credentials the gcloud CLI uses, so a command that works in your terminal proves nothing about your app. In Cloud Run, the service identity needs the role.

echo "GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS"
ls -l ~/.config/gcloud/application_default_credentials.json

2Fix

Grant the role to the identity your code uses, or change which credential ADC finds: unset a stale GOOGLE_APPLICATION_CREDENTIALS, or run gcloud auth application-default login with an account that holds the role.

unset GOOGLE_APPLICATION_CREDENTIALS
gcloud auth application-default login

3Verify

Run your app again, not just gcloud, and read the secret through the client library.

node read-secret.js

Confirmed by docs.cloud.google.com

Did this fix it?

Cause 2 of 4

The role does not include the permission, or sits on another secret or project

1Check

Reading a secret payload needs secretmanager.versions.access. Secret Manager Admin and Secret Manager Secret Accessor include it, while roles/editor and roles/viewer do not. A binding on secret A gives nothing on secret B, and the project in the resource name projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION must be the project where you granted.

gcloud secrets get-iam-policy SECRET_ID
grep -rn "secrets/" src/

2Fix

Bind Secret Manager Secret Accessor to the service account on the exact secret your code reads, at the lowest level that works.

gcloud secrets add-iam-policy-binding SECRET_ID \
  --member="serviceAccount:app-sa@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

3Verify

The policy of that secret now lists the member with the role.

gcloud secrets get-iam-policy SECRET_ID

Confirmed by docs.cloud.google.com

Did this fix it?

Cause 3 of 4

The VM or GKE node does not have the cloud-platform access scope

1Check

For workloads on Compute Engine or GKE, the underlying instance or node must have the cloud-platform OAuth scope. A VM's access scopes cap what the tokens from the metadata server may do, whatever the IAM roles say. The scopes are listed under serviceAccounts in the VM description.

gcloud compute instances describe VM_NAME

2Fix

Stop the VM, set its service account with the cloud-platform scope, and start it again. The VM must be stopped before its scopes can change.

gcloud compute instances stop VM_NAME
gcloud compute instances set-service-account VM_NAME \
  --service-account=SERVICE_ACCOUNT_EMAIL \
  --scopes=cloud-platform
gcloud compute instances start VM_NAME

3Verify

Run the failing call again from the VM.

gcloud secrets versions access latest --secret=SECRET_ID

Confirmed by docs.cloud.google.com

Did this fix it?

Cause 4 of 4

The grant was made a moment ago and has not propagated

1Check

An edit to an IAM policy usually takes about two minutes to propagate and can take 7 minutes or longer. During that time a new grant may not apply yet.

gcloud secrets get-iam-policy SECRET_ID

2Fix

Wait out the propagation window, about two minutes in the usual case and 7 minutes or more at worst, then run the call again. Change nothing in the policy meanwhile.

sleep 420 && node read-secret.js

3Verify

The read succeeds without any further change to the policy.

node read-secret.js

Confirmed by docs.cloud.google.com

Did this fix it?

How do I stop it coming back?

  • Grant Secret Accessor on each secret to the exact service account the workload runs as, and write that binding in your infrastructure code.
  • Give each service its own service account instead of sharing the default one, so a policy tells you who reads what.
  • Always use the same resource name format (projects/PROJECT_ID/secrets/SECRET_ID/versions/latest) from one config value.
  • Log the secret name, not its value, when a read fails, and run the first read at startup so a bad grant shows at deploy time.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Back end

ng serve says it cannot determine the project or target

Cannot determine project for command.

The Angular CLI looked through angular.json for a project that defines the target you asked for (serve, build, test) and found none, or found several and could not pick one.

3 causes, each with a fix

Back end

ng serve cannot find the dev-server builder package

Could not find the implementation for builder @angular/build:dev-server

The builder named in angular.json (for example @angular-devkit/build-angular:dev-server) is looked up as an npm package starting from the workspace root.

3 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.