GKE pods stuck in ErrImagePull with 'failed to fetch oauth token' and 403 or 401
Failed to authorize: failed to fetch oauth token: unexpected status: 403 ForbiddenShort answer
The account that pulls the image is not allowed to read the repository. With 403 the pulling service account lacks the Artifact Registry Reader role, or a VPC Service Controls perimeter blocks it. With 401 the node service account is disabled or the node has no storage read scope. Read the pod event first.
Fix it fast
1Find out which cause you have
Read the pod's events. The last lines give the exact status code and the registry host.
kubectl describe pod POD_NAME -n NAMESPACE2Match what you see, then jump to its fix
- 1
403 Forbidden in the event textThe pulling service account has no read access to the repositoryFix - 2
403 Forbidden although the Reader role is already granted, in a project that uses VPC Service ControlsA VPC Service Controls perimeter blocks Artifact RegistryFix - 3
401 Unauthorized in the event textThe node service account is disabled, or the node lacks a storage read scopeFix
- 1
What does the error look like?
Failed to authorize: failed to fetch oauth token: unexpected status: 403 ForbiddenFailed to authorize: failed to fetch oauth token: unexpected status: 401 UnauthorizedPulling from host HOST_NAME failed with status code: 403 Forbidden
Cause 1 of 3
The pulling service account has no read access to the repository
1Check
Work out which account pulls. If the pod uses an imagePullSecret, it is the service account behind that Secret. Otherwise it is the node pool's service account. Find the node pool's one with this command.
gcloud container clusters describe CLUSTER_NAME \
--location=LOCATION \
--format="table(nodePools.name,nodePools.config.serviceAccount)"2Fix
Grant that account the Artifact Registry Reader role on the repository.
gcloud artifacts repositories add-iam-policy-binding REPOSITORY_NAME \
--location=REPOSITORY_LOCATION \
--member=serviceAccount:SERVICE_ACCOUNT_EMAIL \
--role="roles/artifactregistry.reader"3Verify
Delete the stuck pod so the controller creates a new one, and watch it start.
kubectl delete pod POD_NAME -n NAMESPACE
kubectl get pods -n NAMESPACE -wConfirmed by docs.cloud.google.com
Cause 2 of 3
A VPC Service Controls perimeter blocks Artifact Registry
1Check
Ask whoever runs the organization's perimeters whether the cluster's project and the repository's project are both allowed to reach Artifact Registry.
gcloud access-context-manager perimeters list2Fix
Update the service perimeter so it allows access to Artifact Registry for the pulling project.
# change the perimeter's rules in Cloud console or with gcloud access-context-manager3Verify
Delete the stuck pod and confirm the new one pulls.
kubectl get pods -n NAMESPACE -wConfirmed by docs.cloud.google.com
Cause 3 of 3
The node service account is disabled, or the node lacks a storage read scope
1Check
Pulls by the kubelet use the VM's service account. Check that it is not disabled, and that the node's scopes include devstorage.read_only or cloud-platform.
gcloud iam service-accounts list --filter="email:SERVICE_ACCOUNT_NAME AND disabled:true" \
--project=PROJECT_ID
gcloud compute instances describe NODE_NAME \
--zone="COMPUTE_ZONE" \
--format="flattened(serviceAccounts[].scopes)"2Fix
Re-enable a disabled account. Scopes cannot change on existing nodes, so create a new node pool with the gke-default scopes and move the workloads to it.
gcloud container node-pools create NODE_POOL_NAME \
--cluster=CLUSTER_NAME \
--location=CONTROL_PLANE_LOCATION \
--scopes="gke-default"3Verify
The new nodes list the required scope and pods pull from the repository.
kubectl get pods -n NAMESPACE -o wideConfirmed by docs.cloud.google.com
How do I stop it coming back?
- Grant roles/artifactregistry.reader to a dedicated node service account when the cluster is created.
- Keep one service account per pull path and write down which repository it reads.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Related fixes
Servers
Kubernetes pod stuck in ErrImagePull or ImagePullBackOff
ImagePullBackOff
The kubelet could not pull the container image.
3 causes, each with a fixServers
Status code 460 on an AWS Application Load Balancer
elb_status_code 460 in ALB access logs
A 460 is generated by the load balancer, not by your app.
3 causes, each with a fixServers
Application Load Balancer returns 504 for requests to EC2 instances
HTTPCode_ELB_504_Count
A 504 from an Application Load Balancer means it could not get an answer from a target in time.
4 causes, each with a fixServers
apk cannot find py-pip in an Alpine Docker image
py-pip (no such package):
On current Alpine releases there is no package called py-pip.
2 causes, each with a fixServers
dnf cannot install MySQL on Amazon Linux 2023
dnf cannot find a mysql-server package on Amazon Linux 2023
Amazon Linux 2023 documents MariaDB, not MySQL, as its database server, with packages named mariadb105 and mariadb105-server.
3 causes, each with a fixServers
ng serve runs in the container but localhost:4200 does not open
The browser cannot open localhost:4200 although ng serve is running in the container
ng serve listens on localhost by default, which inside a container is the container itself.
3 causes, each with a fix