Google Secret Manager says access denied although the roles look right
PERMISSION_DENIED when accessing a secret version (permission secretmanager.versions.access)Short answer
Secret Manager checks one thing: does the identity that makes the call hold secretmanager.versions.access on that secret? When the role looks right, the call usually runs as a different identity, the role is on another secret or project, the role lacks the permission, a VM scope blocks it, or the grant is still propagating.
Fix it fast
1Find out which cause you have
Show who has access to the secret you read. Then compare that list with the identity your code really runs as, which is often not the one you granted.
gcloud secrets get-iam-policy SECRET_ID2Match what you see, then jump to its fix
- 1
The policy lists a service account or user with Secret Accessor, but the call still fails.The code runs as a different identity than the one you grantedFix - 2
The principal holds Editor, Viewer or a role on the project, but the policy of this secret does not list Secret Accessor.The role does not include the permission, or sits on another secret or projectFix - 3
The workload runs on a Compute Engine VM, the right service account holds the role, and the call is still denied.The VM or GKE node does not have the cloud-platform access scopeFix - 4
The policy is correct, and the same call succeeds a few minutes later without any change.The grant was made a moment ago and has not propagatedFix
- 1
What does the error look like?
PERMISSION_DENIED when accessing a secret version (permission secretmanager.versions.access)The request failed because the instance could not start successfully.
Cause 1 of 4
The code runs as a different identity than the one you granted
1Check
Application Default Credentials pick a credential in a fixed order: the file named by GOOGLE_APPLICATION_CREDENTIALS, then the file made by gcloud auth application-default login, then the attached service account. The ADC credentials are separate from the credentials the gcloud CLI uses, so a command that works in your terminal proves nothing about your app. In Cloud Run, the service identity needs the role.
echo "GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS"
ls -l ~/.config/gcloud/application_default_credentials.json2Fix
Grant the role to the identity your code uses, or change which credential ADC finds: unset a stale GOOGLE_APPLICATION_CREDENTIALS, or run gcloud auth application-default login with an account that holds the role.
unset GOOGLE_APPLICATION_CREDENTIALS
gcloud auth application-default login3Verify
Run your app again, not just gcloud, and read the secret through the client library.
node read-secret.jsConfirmed by docs.cloud.google.com
Cause 2 of 4
The role does not include the permission, or sits on another secret or project
1Check
Reading a secret payload needs secretmanager.versions.access. Secret Manager Admin and Secret Manager Secret Accessor include it, while roles/editor and roles/viewer do not. A binding on secret A gives nothing on secret B, and the project in the resource name projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION must be the project where you granted.
gcloud secrets get-iam-policy SECRET_ID
grep -rn "secrets/" src/2Fix
Bind Secret Manager Secret Accessor to the service account on the exact secret your code reads, at the lowest level that works.
gcloud secrets add-iam-policy-binding SECRET_ID \
--member="serviceAccount:app-sa@PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/secretmanager.secretAccessor"3Verify
The policy of that secret now lists the member with the role.
gcloud secrets get-iam-policy SECRET_IDConfirmed by docs.cloud.google.com
Cause 3 of 4
The VM or GKE node does not have the cloud-platform access scope
1Check
For workloads on Compute Engine or GKE, the underlying instance or node must have the cloud-platform OAuth scope. A VM's access scopes cap what the tokens from the metadata server may do, whatever the IAM roles say. The scopes are listed under serviceAccounts in the VM description.
gcloud compute instances describe VM_NAME2Fix
Stop the VM, set its service account with the cloud-platform scope, and start it again. The VM must be stopped before its scopes can change.
gcloud compute instances stop VM_NAME
gcloud compute instances set-service-account VM_NAME \
--service-account=SERVICE_ACCOUNT_EMAIL \
--scopes=cloud-platform
gcloud compute instances start VM_NAME3Verify
Run the failing call again from the VM.
gcloud secrets versions access latest --secret=SECRET_IDConfirmed by docs.cloud.google.com
Cause 4 of 4
The grant was made a moment ago and has not propagated
1Check
An edit to an IAM policy usually takes about two minutes to propagate and can take 7 minutes or longer. During that time a new grant may not apply yet.
gcloud secrets get-iam-policy SECRET_ID2Fix
Wait out the propagation window, about two minutes in the usual case and 7 minutes or more at worst, then run the call again. Change nothing in the policy meanwhile.
sleep 420 && node read-secret.js3Verify
The read succeeds without any further change to the policy.
node read-secret.jsConfirmed by docs.cloud.google.com
How do I stop it coming back?
- Grant Secret Accessor on each secret to the exact service account the workload runs as, and write that binding in your infrastructure code.
- Give each service its own service account instead of sharing the default one, so a policy tells you who reads what.
- Always use the same resource name format (projects/PROJECT_ID/secrets/SECRET_ID/versions/latest) from one config value.
- Log the secret name, not its value, when a read fails, and run the first read at startup so a bad grant shows at deploy time.
Sources
- Secret Manager: Access control with IAM
- Secret Manager: Manage access to secrets
- Secret Manager: Access a secret version
- gcloud secrets get-iam-policy
- Authentication: Application Default Credentials
- IAM: Access change propagation
- Compute Engine: Change the attached service account and scopes
- Cloud Run: Configure secrets for services
- Cloud Run: Troubleshoot Cloud Run issues
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Back end
ng serve says it cannot determine the project or target
Cannot determine project for command.
The Angular CLI looked through angular.json for a project that defines the target you asked for (serve, build, test) and found none, or found several and could not pick one.
3 causes, each with a fixBack end
ng fails with Cannot find module @angular/cli/bin/ng
Cannot find module '@angular/cli/bin/ng'
The ng command is the entry point to the Angular CLI package.
2 causes, each with a fixBack end
Angular app shows 'Cannot GET /' when served by a Node.js server
Cannot GET /dashboard
Cannot GET / is Express saying no route or file matches the request.
3 causes, each with a fixBack end
ng serve cannot find the dev-server builder package
Could not find the implementation for builder @angular/build:dev-server
The builder named in angular.json (for example @angular-devkit/build-angular:dev-server) is looked up as an npm package starting from the workspace root.
3 causes, each with a fixBack end
Angular build says it cannot resolve '@angular/localize/init'
Can't resolve '@angular/localize/init' in '/usr/src/app/src'
The build imports @angular/localize, and that package is not in node_modules.
2 causes, each with a fixBack end
The command npm run build -- --prod exited with code 1 (Angular)
The command "npm run build -- --prod" exited with code 1
Exit code 1 only says the build failed; the real message is above it.
2 causes, each with a fix