TechnologyPublished 27 July 2026· Updated 24 August 2026· 5 min read

We Lost DNS Resolution Mid-Cutover and Forgot the Fallback

A 3 AM cutover went dark for 30% of users. Here is the checklist we now run with every VPS-to-AWS migration.

Author

The Cutover That Went Dark: A DNS Migration Horror Story

The Incident: 3 AM and the Site Vanished

Last month we moved a small SaaS client off a random VPS in Singapore and onto AWS Lightsail. The plan looked clean on paper. We had the AMI built, the database dump staged, and the new security groups tested. What we did not have was a working DNS fallback.

At 2:47 AM IST we swapped the NS records at the registrar, moving authority from Enom to Route 53. Within minutes our monitoring dashboards went quiet for Indian traffic. US and European probes were green. By 3:15 AM, 30 percent of users, mostly on Indian ISP resolvers, had lost resolution entirely.

The worst part? We had already decommissioned the old VPS. The snapshot was gone. There was no rollback path, only a rebuild.

What We Tried First (And Why It Failed)

We made three mistakes in a row, each one compounding the last.

First, we swapped NS records at the registrar without lowering TTLs beforehand. The old records sat at 86400 seconds. That meant resolvers could hold the stale parent NS delegation for up to 48 hours, and we had no way to force them to refresh.

Second, we assumed Route 53 would handle propagation instantly. It did not. Route 53 answered authoritatively for the new records, but the parent com zone still pointed a chunk of the world at the old Enom name servers. Those servers were powered off.

Third, we declared victory too early. Monitoring from US-based probes showed 200s across the board. We did not check regional resolvers. Indian ISP resolvers, as it turns out, cache parent NS records aggressively and update slowly. The old NS servers stopped responding, creating a black hole for anyone whose resolver still trusted the stale delegation.

The Working Approach: Shaved TTLs and a Dry-Run Subdomain

We rebuilt the playbook from scratch, and now run it with every migration.

Two weeks before the cutover, we lower all record TTLs from 86400 to 300 seconds. This is the single biggest lever nobody uses. It gives resolvers time to pick up the short TTL before the actual change.

We then do a dry-run migration to a throwaway subdomain on the new host, something like migrate-client.example.com. We sync the files, import the database, point the subdomain, and load the site. If anything breaks, a hardcoded absolute URL, a missing PHP extension, a mod_rewrite rule in the wrong place, we find it here, not on cutover night.

Before the cutover window, we pre-install the Let's Encrypt certificate on the new host using the DNS-01 challenge. HTTP-01 fails if the site is down, so DNS-01 is the only safe option. We use the /etc/hosts trick to pre-issue the cert before DNS cutover.

The cutover window runs 2-3 AM IST, the lowest traffic period for Indian audiences. We update the DNS records, then monitor propagation from 10 global resolvers, not just US-based ones.

We keep the old host snapshot alive for 30 days post-cutover. If anything surfaces later, a missing file, a forgotten subdomain, a broken cron, we can pull from the archive.

Pitfalls We'd Warn an Intern About

Never decommission old infrastructure on cutover day. A team that released the old IP on cutover day converted every subsequent problem from a two-minute reversion into a rebuild.

Parent NS record TTLs can be 48 hours, and you cannot control them. A record-level TTL change does not affect the parent zone NS TTL.

Indian ISP resolvers cache aggressively and update slowly. Monitor DNS propagation from regional resolvers, not just global ones.

Always verify the legacy origin answers correctly before touching DNS. Do not assume it does because nobody turned it off.

TLS cert issuance via HTTP-01 fails if the site is down. Use DNS-01 instead.

Email MX records need a temporary SMTP relay if the registrar is slow to propagate.

What We'd Do Differently Next Time

Lower TTLs at least 48 hours before migration, not 24. The pre-cutover TTL governs the entire recovery budget.

Keep old NS servers alive for 48 hours even after cutover. The reversion is not the end of the incident, you may be retrying the migration soon.

Set up a temporary SMTP relay before touching MX records. Route any straggler traffic back to the new stack.

Use the /etc/hosts trick to pre-issue certs on the new host. This avoids the chicken-and-egg problem of HTTP-01 on a site that is not yet live.

Stage retirement: stop the app first, keep the host and IP allocated longer. Do not release the old IP on cutover day.

Monitor DNS propagation from regional resolvers, not just global ones. A resolver in Mumbai may behave very differently from one in Frankfurt.

Have a documented rollback playbook with exact revert commands ready. When a cutover fails, reverting DNS is the bluntest and most reliable way to send all traffic back to the legacy origin.

The Playbook We Run Now

Here is the checklist we hand to every intern before they touch a production DNS record.

# Two weeks before
# Lower all TTLs to 300 seconds
aws route53 change-resource-record-sets \
  --hosted-zone-id ZXXXXXXXXXXXX \
  --change-batch file://ttl-shave.json

# Dry-run migration
rsync -avz /var/www/ user@new-host:/var/www/
mysqldump -u root -p old_db | mysql -u root -p new_db

# Pre-issue cert with DNS-01
certbot certonly --manual --preferred-challenges dns \
  -d migrate-client.example.com

# Cutover window: 2-3 AM IST
# Update NS records at registrar
# Monitor from 10 global resolvers

The fix took us six hours that night. We had to spin the old servers back up, point DNS back, wait for propagation, then redo the whole migration properly. DNS is the one thing every engineer thinks is simple until it is not.

The lesson is simple: never migrate DNS without lowering TTLs first. And never decommission old infrastructure the same weekend you cut over.