EC2 Ubuntu SSH “Permission denied (publickey)”: 2026 Recovery Runbook

2026 maintenance note: Permission denied (publickey) is an authentication result, not a diagnosis. This runbook starts with read-only evidence, separates client, network, and server causes, and uses AWS-supported recovery paths before offline disk repair. Use it only on instances you are authorized to administer.

Table of Contents

Guardrails and recovery gates

Record the instance ID, Region, Availability Zone, AMI ID, expected OS user, key-pair name, current addresses, root-volume ID, block-device mapping, encryption/KMS state, and DeleteOnTermination value before changing anything.
Open a maintenance window and identify the rollback owner. Snapshot the EBS root volume before any offline write. A snapshot is not a backup until its restore path and KMS access are understood.
Never paste a private key, passphrase, complete authorized_keys file, session token, or unredacted ssh -vvv log into a ticket or chat. Do not disable host-key checking.
Make one bounded change, open a second SSH session to verify it, and keep the recovery channel open until rollback is no longer needed.
Stop if the target instance, account, Region, volume, filesystem, expected user, key fingerprint, or KMS authority is uncertain.

What the error does—and does not—prove

Permission denied (publickey) normally means that a TCP connection reached an SSH service and none of the offered public keys was accepted for that user. A timeout, DNS failure, Connection refused, failed host-key verification, or failed EC2 status check belongs to a different branch. Do not rotate keys or edit server files until the branch is clear.

Capture the AWS-side identity without secrets:

aws ec2 describe-instances \
  --instance-ids i-REPLACE_WITH_INSTANCE_ID \
  --query 'Reservations[0].Instances[0].{State:State.Name,ImageId:ImageId,KeyName:KeyName,AZ:Placement.AvailabilityZone,PublicIp:PublicIpAddress,PrivateIp:PrivateIpAddress,SubnetId:SubnetId,VpcId:VpcId,SecurityGroups:SecurityGroups[*].GroupId,RootDevice:RootDeviceName}' \
  --output yaml

The EC2 KeyName records the key selected at launch; it does not prove that the current authorized_keys file is unchanged. An image, configuration-management run, user-data script, or administrator may have changed it later.

1. Verify the client identity before touching AWS

For an official Ubuntu AMI, the initial user is commonly ubuntu; another AMI or a later-created account can differ. Confirm the AMI and intended account rather than cycling through usernames. Confirm that -i points to the private key, not its .pub file, and restrict the local private-key file as AWS requires:

chmod 400 /secure/path/to/private-key.pem
ssh-keygen -lf /secure/path/to/private-key.pem
ssh-keygen -y -f /secure/path/to/private-key.pem | ssh-keygen -lf -

The last command derives only the public half in memory and prints its fingerprint. It does not reveal the private key. Through an already trusted recovery channel, compare that fingerprint with the fingerprint of the exact line expected for the target user:

sudo -u ubuntu ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys

A matching algorithm name is not enough; the fingerprint must match an authorized line. AWS key-pair fingerprints have source- and algorithm-specific calculation rules, so use AWS’s official fingerprint procedure when comparing with the EC2 console. If the private key is lost, it cannot be downloaded again from EC2; move to a supported recovery route instead of inventing a replacement identity.

2. Verify the server host key

The host key authenticates the server; it is separate from the user key that authenticates you. Obtain the instance fingerprint from a trusted AWS control-plane path and compare it with the SSH prompt:

aws ec2 get-console-output \
  --instance-id i-REPLACE_WITH_INSTANCE_ID \
  --query Output \
  --output text

Look for BEGIN SSH HOST KEY FINGERPRINTS in the system log. Console output can contain hostnames, addresses, boot messages, or user-data output, so sanitize it before sharing. If the presented host key changed unexpectedly, stop: confirm that DNS/IP still identifies this instance and investigate the replacement or rebuild. Never disable strict host-key checking or discard known-host records to work around the warning.

3. Collect a minimal verbose trace

Force the intended identity so an agent loaded with many keys does not obscure the result:

ssh -vvv \
  -o IdentitiesOnly=yes \
  -i /secure/path/to/private-key.pem \
  ubuntu@REPLACE_WITH_VERIFIED_HOSTNAME

Read the trace locally:

Evidence	Likely branch
No route, timeout, or refusal before SSH identification	Address, route, security group, NACL, local/host firewall, or `sshd` availability
Host-key mismatch	Wrong endpoint, rebuilt host, stale trusted record, or possible interception; verify before proceeding
Intended key is never offered	Wrong `-i`, unreadable/unsupported key, SSH config, or agent selection
Intended key is offered and rejected	Wrong OS user, missing/mismatched authorized key, ownership/mode, effective `sshd` policy, or external key command

Verbose logs and fingerprint output can disclose usernames, local paths, IPs, host-key and public-key fingerprints, key comments, proxy commands, and configuration. Redact those fields; never include private-key material or a passphrase.

4. Prove the network path independently

First confirm that the instance is running and both system and instance status checks are healthy. Then inspect—not broaden—the actual path:

aws ec2 describe-instance-status \
  --instance-ids i-REPLACE_WITH_INSTANCE_ID \
  --include-all-instances
aws ec2 describe-security-groups \
  --group-ids sg-REPLACE_WITH_GROUP_ID
aws ec2 describe-network-acls \
  --filters Name=association.subnet-id,Values=subnet-REPLACE_WITH_SUBNET_ID
aws ec2 describe-route-tables \
  --filters Name=association.subnet-id,Values=subnet-REPLACE_WITH_SUBNET_ID

Check the destination address, subnet route, internet gateway/NAT/VPN/bastion or EC2 Instance Connect Endpoint path, security-group source, NACL return traffic, corporate/local egress, and host firewall. Limit SSH ingress to the required source or AWS-managed prefix list; do not open port 22 to the world as a diagnostic shortcut. A healthy VPC path does not prove sshd is listening, and a rejected public key usually indicates that the network path already worked.

5. Prefer an AWS-supported recovery channel

Choose the first route whose prerequisites were already in place. Do not weaken SSH merely to create a recovery route.

Route	Use when	Important gate
EC2 Instance Connect	The AMI/package, IAM permission, username, and network path are supported	It still depends on the instance-side service/configuration and the relevant SSH path or endpoint
Systems Manager Session Manager	The instance is already a managed node with SSM Agent, an appropriate instance profile, and service connectivity	Do not attach broad IAM permissions ad hoc; follow the organization’s Session Manager access policy
EC2 Serial Console	Account/Region, instance type, IAM, and OS login prerequisites are satisfied	It bypasses the VPC data path, but interactive Linux troubleshooting normally needs a preconfigured password-based OS user
`AWSSupport-TroubleshootSSH`	You have Systems Manager Automation authority and want AWS’s EC2Rescue checks	Start with `CheckAll`, which is the default read-only action; review scope before `FixAll` or offline remediation

Once inside through a trusted route, collect server-side evidence before mutation:

id ubuntu
namei -l /home/ubuntu/.ssh/authorized_keys
stat -c '%n owner=%U:%G uid=%u gid=%g mode=%a type=%F' \
  /home/ubuntu \
  /home/ubuntu/.ssh \
  /home/ubuntu/.ssh/authorized_keys
sudo -u ubuntu ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys
sudo sshd -t
sudo sshd -T \
  -C user=ubuntu,host=REPLACE_WITH_INSTANCE_HOSTNAME,addr=REPLACE_WITH_CLIENT_IP \
  | grep -E '^(pubkeyauthentication|authorizedkeysfile|authorizedkeyscommand|strictmodes|allowusers|denyusers|allowgroups|denygroups) '
sudo systemctl status ssh --no-pager

Also inspect sanitized journalctl -u ssh records for the attempt. Effective policy can come from sshd_config, included snippets, Match blocks, or an AuthorizedKeysCommand such as EC2 Instance Connect; do not assume only one file matters.

6. Apply one bounded permission repair

OpenSSH StrictModes rejects key authentication when the home directory, .ssh, or authorized_keys can be modified by other users. Ownership must identify the intended account. The earlier namei and stat output is the baseline; if getfacl is installed, capture the ACL too. On a conventional Ubuntu account using the paths below, repair only these three objects—never run recursive chmod or chown over the home tree:

sudo getfacl -p \
  /home/ubuntu \
  /home/ubuntu/.ssh \
  /home/ubuntu/.ssh/authorized_keys
sudo chown ubuntu:ubuntu /home/ubuntu
sudo chmod go-w /home/ubuntu
sudo chown ubuntu:ubuntu \
  /home/ubuntu/.ssh \
  /home/ubuntu/.ssh/authorized_keys
sudo chmod 700 /home/ubuntu/.ssh
sudo chmod 600 /home/ubuntu/.ssh/authorized_keys

These are conventional restrictive modes, not universal instructions for every deployment. Stop and use the effective AuthorizedKeysFile, account database, ACL/SELinux/AppArmor policy, NFS identity mapping, or image documentation when the layout differs. If the correct public-key fingerprint is absent, treat that as a controlled key-replacement task: preserve the current file, approve the new public fingerprint, follow AWS’s official add/replace-key procedure, and test the new key before removing any old entry.

Validate from a second terminal while keeping the recovery session open:

sudo sshd -t
sudo systemctl is-active ssh
ssh -o IdentitiesOnly=yes \
  -i /secure/path/to/private-key.pem \
  ubuntu@REPLACE_WITH_VERIFIED_HOSTNAME

If the test fails, compare the recorded owner/mode/ACL values and restore only the changed objects. Do not reload or restart ssh when only file metadata changed. If you did change SSH configuration, validate with sshd -t before a controlled reload and retain the recovery channel.

7. Root-volume rescue is the last maintenance path

Use offline repair only when the root is EBS-backed and the supported online routes are unavailable. Before stopping, document downtime approval, dependent services, Auto Scaling behavior, public address behavior, all volume mappings, DeleteOnTermination, and any instance-store data. AWS states that stop/start can change a non-Elastic public IPv4 address and erases instance-store data. Snapshot the root volume and confirm that the helper instance is in the same Availability Zone and has permission to use the volume’s KMS key.

After the original instance is stopped and its root EBS volume is safely detached, attach it to a controlled helper as a data volume. Do not guess /dev/sda2: Nitro instances can enumerate EBS devices as changing NVMe names. Match the EC2 volume ID to the device serial and identify the filesystem and partition read-only:

lsblk -o NAME,SERIAL,SIZE,FSTYPE,UUID,MOUNTPOINTS
sudo blkid
sudo file -s /dev/REPLACE_WITH_VERIFIED_DEVICE

Do not run mkfs, an automatic repair, or a forced filesystem check. Set RESCUE_PARTITION only after the console volume ID, NVMe serial, partition table, filesystem, and expected root contents agree. Mount read-only first:

RESCUE_ROOT='/mnt/ec2-root'
RESCUE_PARTITION='/dev/REPLACE_WITH_VERIFIED_ROOT_PARTITION'
sudo install -d -m 700 "$RESCUE_ROOT"
sudo mount -o ro "$RESCUE_PARTITION" "$RESCUE_ROOT"
findmnt "$RESCUE_ROOT"
sudo test -f "$RESCUE_ROOT/etc/os-release"
sudo test -d "$RESCUE_ROOT/home/ubuntu/.ssh"

Resolve ownership from the offline system’s own account database—not the helper’s ubuntu account—and record metadata before writing:

TARGET_USER='ubuntu'
TARGET_UID="$(awk -F: -v u="$TARGET_USER" '$1==u {print $3}' "$RESCUE_ROOT/etc/passwd")"
TARGET_GID="$(awk -F: -v u="$TARGET_USER" '$1==u {print $4}' "$RESCUE_ROOT/etc/passwd")"
printf 'target uid=%s gid=%s\n' "$TARGET_UID" "$TARGET_GID"
sudo stat -c '%n uid=%u gid=%g mode=%a type=%F' \
  "$RESCUE_ROOT/home/$TARGET_USER" \
  "$RESCUE_ROOT/home/$TARGET_USER/.ssh" \
  "$RESCUE_ROOT/home/$TARGET_USER/.ssh/authorized_keys"
sudo ssh-keygen -lf "$RESCUE_ROOT/home/$TARGET_USER/.ssh/authorized_keys"

If the expected user, UID/GID, fingerprint, or paths do not match, remount nothing read-write and escalate. If they do match and metadata is the proven cause, authorize one bounded write set:

sudo mount -o remount,rw "$RESCUE_ROOT"
sudo chown "$TARGET_UID:$TARGET_GID" "$RESCUE_ROOT/home/$TARGET_USER"
sudo chmod go-w "$RESCUE_ROOT/home/$TARGET_USER"
sudo chown "$TARGET_UID:$TARGET_GID" \
  "$RESCUE_ROOT/home/$TARGET_USER/.ssh" \
  "$RESCUE_ROOT/home/$TARGET_USER/.ssh/authorized_keys"
sudo chmod 700 "$RESCUE_ROOT/home/$TARGET_USER/.ssh"
sudo chmod 600 "$RESCUE_ROOT/home/$TARGET_USER/.ssh/authorized_keys"
sync
sudo mount -o remount,ro "$RESCUE_ROOT"

Verify the resulting numeric owner, modes, fingerprint, and mount state. Unmount cleanly, detach from the helper, and reattach to the original instance using the original block-device mapping recorded before maintenance. Confirm the volume state and attachment before starting. Keep the snapshot and metadata record until status checks, host fingerprint, SSH login, application health, and monitoring all pass; rollback means stopping again and restoring the known-good volume/snapshot under the same change controls.

Stop and escalate when

the error branch changes, the host fingerprint cannot be independently verified, or the intended private/public fingerprints do not match;
the instance is not EBS-backed, instance-store data is at risk, or the target/root device is ambiguous;
the encrypted volume cannot be attached with the approved KMS authority;
filesystem corruption is suspected, the root mounted unexpectedly, or a write would affect more than the approved account paths;
managed policy, centralized identity, configuration management, or Auto Scaling would overwrite the repair.

Official references

2017 original archive (not current instructions)

The complete visible source_export body is preserved below with trailing whitespace normalized. One unsafe recursive permission command is narrowly replaced by [REDACTED: unsafe recursive chmod command]; no other wording or command is changed. The archive contains guessed device names and broad permission advice and must not be used as a current runbook.


N.B. There are dozens of reasons to lead to this problem, for example, wrong permission of your pem file, incorrect username(e.g., ec2-user, ubuntu),wrong spelling in your command, etc..

The reason which causes my problem, if I am right, is that I run command `[REDACTED: unsafe recursive chmod command]` under the wrong directory, namely, my home folder.

The solution is just setting your home folder permissions back.

1. Stop your problematic instance.
2. Create a new instance and stop the new problem-free instance.
3. The newly created instance should be in the same `Availability Zone` like ‘us-west-2c’ which can be set on the ‘Network’ step under which the menu is ‘Subnet’.
4. Detach your ‘ebs volume’ from the problematic instances and attach it on your new problem-free instance.
5. Your need input your instance id as well as the mount point which looks like ‘/dev/sda2’.
6. Start your new instance and mount the second drive that you just attached.

[ubuntu ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdf 202:80 0 100G 0 disk
xvda1 202:1 0 8G 0 disk /
[ubuntu ~]$ sudo mount /dev/xvdf/ /mnt


8. Change directory to your mounted point and restore the permission of your home directory with permission 755.

[ubuntu ~]$ cd /mnt/home/
[ubuntu ~]$ chmod 755 yourusername


9. Stop your new instance and detach the volume owned by the problematic instance.
10. Reattach the just detached volume to the default problematic instance.