Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

EC2 instance shows 'Instance reachability check failed' and will not answer

ErrorInstance reachability check failed

Short answer

A failed instance reachability check means the operating system inside the instance is not responding, so the problem is yours to fix, not AWS hardware. Reboot, then read the system log. Out of memory, a broken root filesystem, SELinux in enforcing mode and a wrong instance type each leave a different line there.

  • Applies to Amazon Linux 2023, Ubuntu 24.04, AWS CLI v2
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Reboot the instance once (this clears old noise from the log), then pull the console output. The last lines before the failure name the cause. Match them to the causes below.

    aws ec2 get-console-output --instance-id i-1234567890abcdef0 --latest --output text
  2. 2Match what you see, then jump to its fix

    1. 1Out of memory: kill processThe instance ran out of memoryFix
    2. 2General error mounting filesystems, or fsck: No such file or directory while trying to openThe root filesystem is damaged, or /etc/fstab asks for a disk that is missingFix
    3. 3Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.SELinux was switched to enforcing and the policy will not loadFix
    4. 4Status checks started failing right after an instance type changeYou changed to a Nitro instance type without the ENA and NVMe driversFix

What does the error look like?

  • Instance reachability check failed
  • Out of memory: kill process
  • General error mounting filesystems
  • Unable to load SELinux Policy. Machine is in enforcing mode. Halting now.

Cause 1 of 4

The instance ran out of memory

1Check

The log shows the kernel killing a process such as httpd or php-cgi. The AWS page names exhausted memory as the cause.

aws ec2 get-console-output --instance-id i-1234567890abcdef0 --latest --output text

2Fix

A reboot brings the instance back, but the AWS page warns the problem will probably return. For an EBS-backed instance, stop it, change it to a larger or memory-optimized instance type, and start it again.

aws ec2 stop-instances --instance-ids i-1234567890abcdef0
# change the instance type in the console: Actions > Instance settings > Change instance type
aws ec2 start-instances --instance-ids i-1234567890abcdef0

3Verify

Both status checks report passed.

aws ec2 describe-instance-status --instance-ids i-1234567890abcdef0

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 2 of 4

The root filesystem is damaged, or /etc/fstab asks for a disk that is missing

1Check

The log ends in a maintenance prompt that EC2 cannot show you. Typical reasons are a detached or failed EBS volume, a corrupted filesystem, or an fstab line for a device that is no longer attached.

aws ec2 get-console-output --instance-id i-1234567890abcdef0 --latest --output text

2Fix

For an EBS-backed instance: stop it, detach the root volume, attach it to a healthy instance in the same Availability Zone, run a filesystem check, fix /etc/fstab if it names a missing device, then reattach the volume as the root device and start the instance. Per the AWS notes, a nonzero sixth fstab field makes boot depend on fsck succeeding, so use it with care.

lsblk
sudo fsck -a /dev/xvdf1

3Verify

After reattaching and starting, the reachability check goes to passed.

aws ec2 describe-instance-status --instance-ids i-1234567890abcdef0

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 3 of 4

SELinux was switched to enforcing and the policy will not load

1Check

The log shows a kernel panic right after that line, so the instance halts during boot.

aws ec2 get-console-output --instance-id i-1234567890abcdef0 --latest --output text

2Fix

Stop the instance, detach its root volume and mount it on a recovery instance. On many systems you disable SELinux by setting SELINUX=disabled in the selinux file under the mounted volume (for example etc/sysconfig/selinux); other distributions differ, so check yours. Unmount, reattach the volume to the original instance and start it.

sudo mount /dev/xvdf1 /mnt/tempvol
sudoedit /mnt/tempvol/etc/sysconfig/selinux   # SELINUX=disabled
sudo umount /mnt/tempvol

3Verify

The instance boots and the status checks pass.

aws ec2 describe-instance-status --instance-ids i-1234567890abcdef0

Confirmed by docs.aws.amazon.com

Did this fix it?

Cause 4 of 4

You changed to a Nitro instance type without the ENA and NVMe drivers

1Check

The AWS page says that after a move to a Nitro-based type, instance status checks fail if the old instance did not have the required ENA and NVMe drivers.

aws ec2 describe-instances --instance-id i-1234567890abcdef0

2Fix

Stop the instance, set the previous instance type again, start it, install the drivers the compatibility page asks for, and only then change the type.

aws ec2 stop-instances --instance-ids i-1234567890abcdef0
# set the previous instance type in the console, then:
aws ec2 start-instances --instance-ids i-1234567890abcdef0

3Verify

The instance passes both status checks on the old type.

aws ec2 describe-instance-status --instance-ids i-1234567890abcdef0

Confirmed by docs.aws.amazon.com

Did this fix it?

How do I stop it coming back?

  • Create a CloudWatch alarm on the status check so a failure reaches you before a user reports it.
  • Snapshot EBS volumes on a schedule; a failed volume is then a restore, not a loss.
  • Check every /etc/fstab edit before you reboot a production server, and leave the sixth field at 0 for disks that may be detached.
  • Watch memory use and size the instance with headroom.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Servers

Apache has a certificate, but https:// does not open

No line with :443 in the output of ss -ltnp

HTTPS needs four things at once: mod_ssl loaded, Apache listening on 443, a virtual host on 443 with SSLEngine on, and a firewall that lets 443 in.

4 causes, each with a fix

Servers

AWS CLI says you must specify a region

You must specify a region. You can also configure your region by running "aws configure".

The AWS CLI sends every request to one region and has none to use.

4 causes, each with a fix

Servers

EC2 Instance Connect from the browser will not open an SSH session

Host key validation failed for EC2 Instance Connect

Browser-based EC2 Instance Connect needs five things at once: a fresh host key in the AWS database, a current agent package, port 22 open to the EC2 Instance Connect prefix list, a public address with a route (or an Instance Connect Endpoint), and a browser that can reach the proxy on port 443.

5 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.