SSH Key Rotation Broke Our VPS Cutover at 3 AM

The 3 AM Cutover That Went Dark

We were migrating a client off a budget VPS provider at 2:45 AM IST. The old server was in read-only mode, DNS TTLs were shaved to 300 seconds, and the final rsync was running. At 2:52 AM, SSH stopped accepting connections from our jump host. The error was Permission denied (publickey) despite using the same key pair that worked 20 minutes earlier.

This was not a permissions issue. It was not a username issue. It was not a firewall issue. It was the VPS provider silently rotating SSH host keys during an unattended upgrade window.

The Incident: SSH Key Rotation During VPS Migration

The migration playbook we follow is borrowed from VillageHosting.in's zero-downtime approach: shave DNS TTLs to 300 seconds, do a dry run 24 hours ahead, sync with rsync -av --delete, put the old site in read-only mode, then flip DNS and validate from multiple global resolvers How we do zero-downtime migrations.

That night, we were on Step 3 of the final cutover. The old server had been in read-only mode for 15 minutes. The rsync was almost done. Then SSH died.

What We Tried and What Failed

Attempt 1: Key File Permissions

We checked the local key file permissions first, assuming the classic mistake. ls -la ~/.ssh/migration_key.pem showed 600, which should be fine. We ran chmod 400 anyway and retried. Still failed.

Attempt 2: Wrong Username

We cycled through common usernames: root, admin, ubuntu, ec2-user. None worked. The VPS provider used a custom username vpsuser that we had to look up in their documentation.

Attempt 3: Security Group / Firewall

We verified the cloud firewall rules allowed our IP. They did. We temporarily opened SSH to 0.0.0.0/0. Still no connection.

Attempt 4: Regenerate and Re-upload Key

We generated a new key pair, uploaded the public key to the VPS control panel, and tried again. The control panel showed the key was saved, but SSH still rejected us.

The Root Cause: Host Key Rotation by the Provider

Unattended Upgrades Triggered at 3 AM

At 3 AM, the VPS provider's automated maintenance kicked in. Unattended-upgrades ran on the old server, which included a security update to OpenSSH. This triggered a regeneration of the SSH host keys (/etc/ssh/ssh_host_*_key). Our known_hosts file had the old server fingerprint, causing SSH to refuse the connection as a man-in-the-middle protection.

A ServerFault thread confirms this exact pattern: host keys updated at 3 AM during unattended-upgrades, breaking autossh connections Host Keys updated unexpectedly.

The Real Fix: Update known_hosts

The fix was simple but we wasted 38 minutes on it:

ssh-keygen -R old-server-ip
ssh old-server-ip

This removed the stale host key and re-added the new one after we manually verified the fingerprint matched what the VPS provider showed in their console.

The Working Approach: Inventory, Rehearsal, Rollback

Pre-Migration Inventory

Before any migration, we now run a full inventory script:

#!/bin/bash
echo '=== SSH Host Keys ==='
ssh-keyscan -H $OLD_IP >> ~/.ssh/known_hosts_backup
echo '=== Authorized Keys ==='
scp root@$OLD_IP:/root/.ssh/authorized_keys ./backup_authorized_keys
echo '=== Firewall Rules ==='
scp root@$OLD_IP:/etc/ssh/sshd_config ./backup_sshd_config

This captures the current state so we can detect changes during migration.

Dry Run with TTL Shaving

We schedule the dry run 24 hours before the actual migration:

1. Lower DNS TTL to 300 seconds
2. Sync all files with rsync -av --delete
3. Take a snapshot of the VPS
4. Test SSH access from multiple jump hosts
5. Verify all services start correctly

Rollback Plan

If SSH breaks during migration:

1. Restore VPS from snapshot (takes 2-3 minutes)
2. Revert DNS to old IP
3. Use VPS provider's web-based console (no SSH needed) to fix host keys
4. Re-add our key to authorized_keys via console

Pitfalls We Warn Interns About

Never Trust known_hosts Across Migrations

SSH host key mismatches are the number one cause of 3 AM panic. Always run ssh-keygen -R before connecting to a server that may have been rebuilt or updated.

VPS Provider Maintenance Windows Are Invisible

Budget VPS providers often run unattended upgrades at 3 AM local time. Check their status page and maintenance schedule before scheduling your migration.

The Web Console Is Your Lifeline

When SSH fails, the VPS provider's web-based console (serial console) is the only way in. Test it during the dry run. It bypasses all SSH configuration issues.

Snapshots Are Not Backups

A VPS snapshot captures disk state but not network state. If the provider rotates host keys during a snapshot restore, you still get the same SSH failure.

What We Would Do Differently Next Time

Pre-empt Host Key Changes

We would add a pre-migration step to proactively regenerate and document host keys:

# On old server, before migration window
sudo ssh-keygen -A
sudo systemctl restart sshd
# Document new fingerprints
sudo ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub

Use SSH Certificates Instead of Key Pairs

For future migrations, we would set up an SSH CA and issue short-lived certificates. This eliminates host key mismatch issues entirely since the CA signs both server and client keys.

Schedule Around Provider Maintenance

We would check the VPS provider's maintenance calendar and avoid scheduling migrations during their automated update windows. Even if it means waiting a week for the next low-traffic window.

Sources

• How we do zero-downtime migrations
• Host Keys updated unexpectedly
• Troubleshooting Tuesday: 3 AM SSH failures
• 3am. Production is down. SSH is refusing connections.
• ssh-key migration to Ed25519
• How to Rotate SSH Keys Safely
• SSH Lockout Recovery in Production