Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

GKE pods stuck in ErrImagePull with 'failed to fetch oauth token' and 403 or 401

ErrorFailed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

Short answer

The account that pulls the image is not allowed to read the repository. With 403 the pulling service account lacks the Artifact Registry Reader role, or a VPC Service Controls perimeter blocks it. With 401 the node service account is disabled or the node has no storage read scope. Read the pod event first.

  • Applies to GKE Standard, Artifact Registry, gcloud CLI, kubectl 1.3x
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the pod's events. The last lines give the exact status code and the registry host.

    kubectl describe pod POD_NAME -n NAMESPACE
  2. 2Match what you see, then jump to its fix

    1. 1403 Forbidden in the event textThe pulling service account has no read access to the repositoryFix
    2. 2403 Forbidden although the Reader role is already granted, in a project that uses VPC Service ControlsA VPC Service Controls perimeter blocks Artifact RegistryFix
    3. 3401 Unauthorized in the event textThe node service account is disabled, or the node lacks a storage read scopeFix

What does the error look like?

  • Failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  • Failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
  • Pulling from host HOST_NAME failed with status code: 403 Forbidden

Cause 1 of 3

The pulling service account has no read access to the repository

1Check

Work out which account pulls. If the pod uses an imagePullSecret, it is the service account behind that Secret. Otherwise it is the node pool's service account. Find the node pool's one with this command.

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --format="table(nodePools.name,nodePools.config.serviceAccount)"

2Fix

Grant that account the Artifact Registry Reader role on the repository.

gcloud artifacts repositories add-iam-policy-binding REPOSITORY_NAME \
    --location=REPOSITORY_LOCATION \
    --member=serviceAccount:SERVICE_ACCOUNT_EMAIL \
    --role="roles/artifactregistry.reader"

3Verify

Delete the stuck pod so the controller creates a new one, and watch it start.

kubectl delete pod POD_NAME -n NAMESPACE
kubectl get pods -n NAMESPACE -w

Confirmed by docs.cloud.google.com

Did this fix it?

Cause 2 of 3

A VPC Service Controls perimeter blocks Artifact Registry

1Check

Ask whoever runs the organization's perimeters whether the cluster's project and the repository's project are both allowed to reach Artifact Registry.

gcloud access-context-manager perimeters list

2Fix

Update the service perimeter so it allows access to Artifact Registry for the pulling project.

# change the perimeter's rules in Cloud console or with gcloud access-context-manager

3Verify

Delete the stuck pod and confirm the new one pulls.

kubectl get pods -n NAMESPACE -w

Confirmed by docs.cloud.google.com

Did this fix it?

Cause 3 of 3

The node service account is disabled, or the node lacks a storage read scope

1Check

Pulls by the kubelet use the VM's service account. Check that it is not disabled, and that the node's scopes include devstorage.read_only or cloud-platform.

gcloud iam service-accounts list --filter="email:SERVICE_ACCOUNT_NAME AND disabled:true" \
  --project=PROJECT_ID
gcloud compute instances describe NODE_NAME \
  --zone="COMPUTE_ZONE" \
  --format="flattened(serviceAccounts[].scopes)"

2Fix

Re-enable a disabled account. Scopes cannot change on existing nodes, so create a new node pool with the gke-default scopes and move the workloads to it.

gcloud container node-pools create NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --scopes="gke-default"

3Verify

The new nodes list the required scope and pods pull from the repository.

kubectl get pods -n NAMESPACE -o wide

Confirmed by docs.cloud.google.com

Did this fix it?

How do I stop it coming back?

  • Grant roles/artifactregistry.reader to a dedicated node service account when the cluster is created.
  • Keep one service account per pull path and write down which repository it reads.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Related fixes

Servers

dnf cannot install MySQL on Amazon Linux 2023

dnf cannot find a mysql-server package on Amazon Linux 2023

Amazon Linux 2023 documents MariaDB, not MySQL, as its database server, with packages named mariadb105 and mariadb105-server.

3 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.