Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

EKS managed node group ends in Create failed because nodes never joined

ErrorInstances failed to join the Kubernetes cluster

Short answer

Nodes that do not register within 15 minutes cause NodeCreationFailure and a Create failed status. AWS lists the common causes: missing node IAM role policies, no outbound path to the cluster, a cluster endpoint that nodes cannot reach, and a VPC without DNS options.

  • Applies to Amazon EKS managed node groups, AWS CLI v2
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Read the health issues of the node group. The code and message narrow the cause before you change anything.

    aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name <nodegroup> --query nodegroup.health
  2. 2Match what you see, then jump to its fix

    1. 1NodeCreationFailure; AWS names insufficient node IAM role permissions as a common causeThe node IAM role is missing required policiesFix
    2. 2NodeCreationFailure on nodes in a public subnet without a public IP, or in a private subnet without a route outThe nodes cannot reach the cluster API serverFix
    3. 3The console shows Instances failed to join the Kubernetes clusterThe cluster endpoint access settings exclude the nodesFix
    4. 4The kubelet log contains: node "" not foundThe VPC has no DNS option values, so nodes have no private DNS entryFix

What does the error look like?

  • Instances failed to join the Kubernetes cluster
  • NodeCreationFailure
  • Create failed

Cause 1 of 4

The node IAM role is missing required policies

1Check

The node role needs AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryPullOnly. List what is attached.

aws iam list-attached-role-policies --role-name <node-role-name>

2Fix

Attach the two managed policies to the node role. The VPC CNI policy AmazonEKS_CNI_Policy belongs on the role used by the aws-node service account, which AWS recommends over putting it on the node role.

aws iam attach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy \
  --role-name <node-role-name>
aws iam attach-role-policy \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly \
  --role-name <node-role-name>

3Verify

Recreate the node group, or wait for the replacement nodes, and check its health.

aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name <nodegroup> --query nodegroup.status

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 4

The nodes cannot reach the cluster API server

1Check

Nodes must either reach the internet through a public IP address (and a security group that allows it), or meet the requirements for private clusters with limited internet access. Check the subnet's route table and whether nodes get a public IP.

aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>

2Fix

Give nodes a route out (public IP in a public subnet, or a NAT path from a private subnet), or build the private-cluster endpoints AWS documents.

# public subnet: make new instances receive a public IPv4 address
aws ec2 modify-subnet-attribute --subnet-id <subnet-id> --map-public-ip-on-launch

3Verify

Create the node group again and wait for the nodes to register.

kubectl get nodes

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 4

The cluster endpoint access settings exclude the nodes

1Check

Make sure that either private endpoint access is enabled for the cluster, or the CIDR blocks allowed on the public endpoint include the addresses the nodes use to reach it.

aws eks describe-cluster --name <cluster> --query cluster.resourcesVpcConfig

2Fix

Enable private access, or add the nodes' outbound addresses to the public access CIDRs.

aws eks update-cluster-config --name <cluster> --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

3Verify

Wait for the update to finish, then create the node group again.

aws eks describe-cluster --name <cluster> --query cluster.resourcesVpcConfig.endpointPrivateAccess

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 4 of 4

The VPC has no DNS option values, so nodes have no private DNS entry

1Check

The VPC needs domain-name and domain-name-servers set in its DHCP options. The defaults are <region>.compute.internal and AmazonProvidedDNS.

aws ec2 describe-dhcp-options

2Fix

Create a DHCP options set with those values and associate it with the VPC.

# DHCP options set values (default):
#   domain-name         = <region>.compute.internal
#   domain-name-servers = AmazonProvidedDNS

3Verify

Launch a new node and watch it register.

kubectl get nodes -w

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Create node roles from the AWS-documented policy set and keep them in infrastructure code.
  • Decide the cluster endpoint access mode before adding nodes, and test one node group before scaling out.
  • Keep spare IP addresses in the node subnets; node creation also fails when a subnet runs out.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.