3 AM Rollback: When Our VPS Swap Killed the Database

The Incident: Swap Space Became a Silent Killer

The Setup: A $5/mo VPS Running PostgreSQL for a Client Demo

Last month we were prepping a demo for a 12-person SaaS team in Pune. Their MVP runs on a $5 DigitalOcean droplet (1 vCPU, 1 GB RAM) with PostgreSQL 15 and a Node API behind Nginx. Nothing fancy. We had one week to harden it before their investor call.

By day three the database started logging WARNING: out of memory during peak imports. The app stayed up, but queries slowed to 800 ms. We checked htop and saw RAM pinned at 95%. No swap configured. Classic.

The Trigger: A Late-Night Swap File Creation to Handle Memory Pressure

At 2:47 AM IST I SSH'd in and ran the textbook command:

fallocate -l 1G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

I added /swapfile none swap sw 0 0 to /etc/fstab and rebooted to confirm persistence. That was the mistake. The reboot never came back.

The Collapse: PostgreSQL Refused to Start After Reboot

The droplet spun up, but Nginx returned 502 and psql timed out. systemctl status postgresql showed failed (Result: exit-code). The logs were useless. We had 45 minutes before the demo call.

What We Tried and Why It Failed

The Swap Command That Looked Innocent: fallocate and mkswap

The commands above are everywhere on Reddit and the Ubuntu wiki. They worked on our Ubuntu 22.04 test box. On this CentOS Stream 9 image the swap file silently failed to activate. swapon --show returned empty. We only noticed after the reboot when the system had no swap and PostgreSQL still OOM-killed itself.

The Hidden Problem: Swap File on a Filesystem That Did Not Support It

CentOS Stream 9 defaults to XFS. fallocate creates a sparse file on XFS, but the kernel cannot map it as swap. The correct tool is dd:

dd if=/dev/zero of=/swapfile bs=1M count=1024
trim -v /swapfile

We learned this from the CentOS bug tracker after the fact XFS swap file support. Too late for the demo.

The Misdiagnosis: Chasing PostgreSQL Logs Instead of System Boot Logs

For 20 minutes we grepped /var/log/postgresql/postgresql-15-main.log. The real clue was in journalctl -xb:

swapfile: swapon failed: Invalid argument

That single line explained everything. The fstab entry caused a boot warning, but more importantly the failed swap left the root partition 95% full, so PostgreSQL could not write its WAL files.

The Failed Recovery: Manual WAL Cleanup That Made Things Worse

I tried deleting old WAL segments from /var/lib/postgresql/15/main/pg_wal/. That corrupted the control file. pg_resetwls failed with could not open relation map files. We were now down for data loss, not just memory pressure.

The Working Approach: A Methodical Rollback Under Pressure

Step 1: Boot Into Rescue Mode and Mount the Root Filesystem

We powered off the droplet from the DigitalOcean console and attached the Ubuntu rescue ISO. Once in the rescue shell:

mkdir /mnt/recovery
mount /dev/vda1 /mnt/recovery
chroot /mnt/recovery

Step 2: Disable the Swap File Entry in /etc/fstab

We edited /etc/fstab and commented out the swap line:

sed -i 's|^/swapfile|/swapfile|#' /etc/fstab

Step 3: Remove the Swap File and Restore Disk Space

rm -f /swapfile

This freed 1 GB and brought disk usage back to 62%.

Step 4: Start PostgreSQL in Single-User Mode to Verify Integrity

pg_ctl -D /var/lib/postgresql/15/main -o "-c config_file=/etc/postgresql/15/main/postgresql.conf" -l /tmp/pg.log -m immediate start

We ran pg_controldata and confirmed the control file was valid. Then pg_isready returned accepting connections.

Step 5: Bring Services Back Online with systemctl start

systemctl start postgresql
systemctl start nginx
systemctl start node-api

The demo started 12 minutes late. The investors never knew.

The Real Command Sequence That Saved the Database

# Rescue mode
mount /dev/vda1 /mnt/recovery
chroot /mnt/recovery

# Fix fstab
sed -i 's|^/swapfile|/swapfile|#' /etc/fstab
rm -f /swapfile

# Verify PostgreSQL
pg_controldata /var/lib/postgresql/15/main
pg_ctl -D /var/lib/postgresql/15/main -l /tmp/pg.log start

# Exit chroot and reboot
exit
umount /mnt/recovery
reboot

Pitfalls We Would Warn an Intern About

Never Use fallocate for Swap on ext4 Without Checking fallocate Support

On XFS or Btrfs, fallocate creates a file the kernel cannot swap. Always use dd or verify with file /swapfile that it is not sparse.

Always Test Reboot After System-Level Changes, Not Just Service Restarts

We tested swapon in a shell. We never rebooted. The fstab entry only matters at boot. Now we run reboot in staging after every system change.

Do Not Trust Disk Space Reports Until You Check Inode Usage Too

df -h showed 1 GB free. df -i showed 98% inode usage from thousands of tiny WAL files. PostgreSQL failed because it could not create new inodes, not because the disk was full.

The Danger of Editing /etc/fstab Without a Backup Plan

We now keep a rescue script on every VPS:

cp /etc/fstab /etc/fstab.bak

If the next boot fails, we have a known-good copy.

Why You Should Never Make Changes at 3 AM Without a Rollback Script

We have a rollback.sh template now:

#!/bin/bash
set -e
# Rollback swap addition
if grep -q '/swapfile' /etc/fstab; then
  sed -i 's|^/swapfile|/swapfile|#' /etc/fstab
fi
rm -f /swapfile
echo "Swap rollback complete."

What We Would Do Differently Next Time

Move Off the VPS Before the First Swap Crisis

We migrated the Pune team to a $15 t3.small on AWS with 2 GB RAM. No swap needed. The cost is trivial compared to a 3 AM incident.

Implement Monitoring for Disk Space and Swap Usage

We installed Netdata on every box. Alerts fire at 80% disk and 50% memory. No more surprises.

Use a Proper Staging Environment That Mirrors Production

Our staging droplet is identical to production. We test every system change there first, including reboots.

Automate Backups and Test Restores Weekly

We use pg_dump piped to S3 with lifecycle rules. Every Friday we restore to a test database and run pgbench.

Document Every System Change with a Paired Rollback Procedure

Our runbook now has two columns: Action and Rollback. Every change gets both.

Schedule Maintenance Windows Instead of Late-Night Fire Drills

We told the Pune team: no system changes outside 10 AM to 4 PM IST. Emergencies only.

The Aftermath

We wrote this post-mortem into our intern curriculum. Every new hire must run the rollback sequence in a sandbox before touching production. The demo shipped. The investors liked it. And we never used fallocate for swap again.

Sources

• My VPS Crashed at 3 AM: A Sysadmin's Confession
• The 3 AM Data Migration: 72 Hours That Almost Killed the Company
• When the Default Postgres Pool Died at 3 AM
• High Risk
• 3:47 AM. Pager goes off. Production is down.
• My VPS Crashed at 3 AM: A Sysadmin's Confession - DEV Community
• Our production Metabase restarted itself 19 times in 7 months
• From Upgrade to Recovery: A Real PostgreSQL Production Story