We Skipped the Pre-Cutover Smoke Test and Paid for It

The Incident: A $440M Lesson from TSB Bank (April 2018)

We still keep a printed copy of the TSB post-mortem pinned above our war room monitor in Sikar. Not because we worked on it, but because it is the cleanest example of what happens when a fixed date becomes the only go/no-go criterion. In April 2018, TSB Bank began moving 5.2 million customer accounts off the Lloyds platform onto infrastructure owned by Spanish parent Sabadell. The migration weekend was politically immovable. There was no rollback path. Within 45 minutes, 1.9 million customers were locked out of their own money. Online banking crashed. Payments failed. Fraud controls misfired. Some customers could see other people's accounts. The outage lasted weeks. Parliament held emergency hearings. The CEO resigned. Total cost: £366M, including £80M in customer redress and roughly 80,000 customers lost.

We did not cause that failure, but we have caused smaller versions of it. In 2025, we helped a 12-person SaaS startup in Pune migrate off a random VPS in Singapore. They had a fixed launch date for a funding demo. We skipped the pre-cutover smoke test. The new origin worked on its IP. It failed under the production hostname because the middleware was stripping the Host header. The demo went down for 90 minutes. No customer data was lost, but the funding round slipped by six weeks.

That is the scale we are talking about here. Not millions of pounds, but real momentum, real trust, and real time. The TSB story is the extreme. Our story is the everyday version that every intern should know how to prevent.

What We Were Migrating

The Pune startup ran a Node.js API behind an Nginx reverse proxy on a $12/month VPS. Their database was a single MySQL instance. They had outgrown the VPS, and their investor wanted to see the app running on AWS with auto-scaling before the next funding call. The migration target was a fresh ECS service behind an Application Load Balancer, with RDS MySQL as the database.

The plan looked simple on paper. Dump the database, spin up the new stack, import the data, flip the DNS. The date was set two weeks out. No one asked whether the new origin would respond correctly to the production hostname. No one asked whether the middleware would pass the Host header through. No one asked whether there was a rollback plan if the import failed halfway through.

That is the TSB pattern in miniature. A fixed date. A one-way door. An assumption that the new environment would behave like the old one.

What We Tried (And What Failed)

The Dry Run That Wasn't Real

We ran a dry run at 10% of user volume. The database dump was small. The import was fast. Everything looked green. But the dry run used a subset of data, not production-equivalent volumes. The real migration would take 4 hours, not 40 minutes. The validation scripts timed out on the full dataset. The smoke tests never ran against the production hostname.

Scale testing is not optional. Failure modes that destroy projects appear at 100%, not 10%. The TSB team tested at 10% of customer load. When 1.9 million people hit the system at once, fraud controls that had never been validated in the new environment locked everyone out.

The Missing Smoke Test

We deployed the new origin. It responded on its IP. We ran curl -k https://NEW_IP/health and saw a 200. We assumed the new origin would work under real traffic conditions. It did not.

An origin that works on its IP may fail under the production hostname. TLS SNI, Host header routing, and middleware stripping only surface in real conditions. We should have generated a preview URL that proxies the real domain through the new origin, then run automated checks: Lighthouse, visual diff, security headers, console-error scan.

The One-Way Door

We had no viable rollback plan once cutover began. The go/no-go decision was driven by the calendar, not readiness criteria. If the import failed, we would have had to rebuild the old VPS from a stale backup. That was not a plan. It was a hope.

The Working Approach: Mock Cutover at T-14

We now run a mock cutover at T-14 days, two weeks before the production cutover. It is a time-boxed, partial rehearsal of the cutover plan. It exercises the high-risk technical phases of the cutover window: source-system lock, final data migration, validation, and target-system smoke tests. It runs against a production-equivalent non-production environment. It stops after smoke tests. It does not proceed to DNS cutover or user-facing changes.

What a Mock Cutover Actually Covers

A mock cutover covers the riskiest phases while there is still time to fix them. Here is what we do:

• Source-system lock procedure in non-prod
• Final backup snapshot and data extraction
• Migration to target environment with production-equivalent data
• Row-count and checksum validation
• Smoke tests against the target using the same scripts as the real cutover

We capture pass/fail and elapsed time for every step. We measure total elapsed time against the production window with a 30% buffer. We measure data migration time as the longest single phase. We measure validation time and smoke test time per step. We measure time-to-recover from a failure and document it.

Real Commands and File Paths

Here are the actual commands we run during a mock cutover:

mysqldump --single-transaction --routines --triggers prod_db | gzip > backup.sql.gz

/opt/migration/validate_row_counts.sh prod_staging_dump.csv target_staging_dump.csv

curl -H "Host: production.example.com" https://NEW_IP/health

/opt/smoke-tests/run_smoke_suite.sh --target https://preview.testurl.live/abc123

The first command creates a consistent backup. The second validates that the row counts match between source and target. The third tests the new origin under the production hostname. The fourth runs the full smoke test suite against a preview URL.

Time Measurement and Metrics

We track five metrics during every mock cutover:

• Total elapsed time: Does the plan fit in the production window with a 30% buffer?
• Data migration time: The longest single phase; drives the rest of the timeline.
• Validation time: Often underestimated; failures here force rollback decisions.
• Smoke test time: If smoke tests take longer than the buffer, the plan needs revision.
• Time-to-recover from a failure: If a step fails, how long does it take to retry and succeed?

Pitfalls We Would Warn an Intern About

Don't Trust the Calendar

A fixed date is not a go/no-go criterion. Readiness is. We write go/no-go criteria before the date is set, not after. Political pressure will always push for an immovable date. We treat the date as a target, not a commitment until readiness is confirmed.

Scale Testing Is Not Optional

The failure modes that destroy projects appear at 100%, not 10%. Test at production-equivalent data volumes in dress rehearsals. Mock cutovers can use small datasets, but dry runs and dress rehearsals cannot.

The Hostname Trap

An origin that works on its IP may fail under the production hostname. TLS SNI, Host header routing, and middleware stripping only surface in real conditions. Use a preview URL proxy (like TestURL.live) to validate end-to-end before DNS flip.

Rollback Is Not a Postscript

If you cannot answer "what do we do if this fails in hour one," you are not ready. A one-way door cutover means no recovery path when things go wrong. Build rollback into the plan, not as an afterthought.

What We Would Do Differently Next Time

Schedule a Mock Cutover at T-14

We run the riskiest phases: data migration, validation, and smoke tests. We stop after smoke tests. We do not proceed to DNS cutover or user-facing changes. We capture pass/fail and elapsed time for every step.

Add a Dress Rehearsal at T-7

We execute the full cutover plan against a representative non-prod environment. Same people, same sequence, same data volumes as production. We confirm that action items from the mock cutover are closed.

Validate the New Origin Before DNS Flip

We deploy the new origin and confirm it responds to its own hostname or IP. We generate a preview URL that proxies the real domain through the new origin. We run automated checks: Lighthouse, visual diff, security headers, console-error scan.

Write Go/No-Go Criteria Before the Date Is Set

We define objective readiness gates independent of the calendar. We require sign-off from engineering, security, and business stakeholders. We treat the date as a target, not a commitment until readiness is confirmed.

Build a Real Rollback Plan

We document the exact steps to revert to the old system. We test rollback in the dress rehearsal, not just the forward path. We ensure rollback does not depend on the old system being untouched for weeks.

The Intern Takeaway

Every intern who joins our lab gets a copy of the TSB post-mortem and the Knight Capital case study. They read about the 8-year-old code that misfired because no post-deployment smoke test was run before market open. They read about the missing kill switch and the undrilled incident response playbook.

Then we make them run a mock cutover. Not a code review. Not a unit test. A real, time-boxed rehearsal of the riskiest phases of a migration. They learn that a fixed date is not a go/no-go criterion. They learn that scale testing is not optional. They learn that an origin that works on its IP may fail under the production hostname.

They learn that rollback is not a postscript. It is the first thing you plan, the first thing you test, and the last thing you skip.

Because the cost of skipping it is not just money. It is trust. It is time. It is the next funding round that slips by six weeks. It is the customer who leaves because their account was locked for 90 minutes.

And sometimes, it is £366M.

Sources

• Mock cutover: an explainer | Cut-over
• DevOps Pre-Cutover Validation, Smoke Test Before DNS Cutover | TestURL.live
• Go-live on a fixed date. No way back. | Surinder Verma, PMP®
• 1 missed server. 8-year-old code. $440M. 45 minutes. | Surinder Verma, PMP®
• Cutover rehearsal: the complete guide for integrators | Cut-over
• From a 13.5-Hour Mock to a 3-Hour Production Cutover - DEV Community
• Designing Enterprise Cutovers That Don't Fail - Andrew Weinberg