Security Group Gaps That Stalled Our Cloud Cutover at 2 AM

The Night Our Cutover Stalled at 2 AM

It was 02:14 IST on a Saturday. The maintenance window for a FinTech startup's migration from a single DigitalOcean droplet to AWS EKS had started at midnight. We had Terraform plans approved, staging validated, and a runbook printed. By 02:14 the cutover was stalled. Stripe webhooks were timing out. New EKS nodes could not reach the RDS cluster. The Auto Scaling Group launch template referenced a security group that had been deleted two days earlier.

We rolled back at 03:47 IST. The postmortem took three days. This article captures the inventory, rehearsal, and rollback steps we now walk through with every startup moving off a random VPS.

The Client Context: A FinTech Startup on a Single DigitalOcean Droplet

The team ran a Node.js API and PostgreSQL on one droplet in Bangalore region. No VPC, no managed database, no load balancer. They processed ~2,000 transactions per day through Stripe. The migration target: EKS in ap-south-1 with RDS PostgreSQL, ALB, and GitHub Actions CI/CD. Timeline: four weeks. Team: two backend engineers, one DevOps contractor, and our two-person platform squad.

We started with a dependency inventory. Not a diagram. A spreadsheet with columns: source, destination, port, protocol, current SG rule, required SG rule, owner, tested in staging. That spreadsheet became the single source of truth for the cutover.

What the Inventory Revealed Before We Touched Anything

Running aws ec2 describe-security-groups --region ap-south-1 | jq '.SecurityGroups[] | {GroupId, GroupName, IpPermissions, IpPermissionsEgress}' against the existing AWS account (set up months earlier for experiments) surfaced three gaps:

1. The EKS node group SG sg-0a1b2c3d4e5f6g7h8 had no egress rule for Stripe's CIDR ranges.
2. The ASG launch template lt-0abcdef1234567890 referenced sg-0deadbeef12345678 which did not exist.
3. The RDS SG sg-0r1d2s3q4l5e6r7s8 allowed ingress only from 10.0.3.0/24, a subnet used by batch jobs, not the new EKS node subnet 10.0.4.0/24.

We found these before the maintenance window because we treated inventory as a deliverable, not a checkbox.

The Incident: Security Group Gaps That Surfaced During Cutover Rehearsal

The Missing Egress Rule That Blocked Stripe Webhooks

During the Thursday rehearsal (two days before cutover) we deployed the payment service to staging EKS and triggered a test Stripe webhook. The request hit the ALB, reached the pod, but the outbound call to api.stripe.com timed out. tcpdump on the node showed SYN packets leaving the pod but no SYN-ACK returning. The node SG egress rules allowed 10.0.0.0/16 and a legacy payment gateway CIDR. Stripe's CIDR was missing. This matches the pattern documented in the October 2024 postmortem where an EKS node group SG update omitted Stripe's CIDR range, causing 47 minutes of payment outage Postmortem: How a Misconfigured Istio 1.22 and AWS Security Group Brought Down Our Payment System.

The ASG Launch Template Referencing a Deleted SG

The same rehearsal triggered a scale-out event when we ran kubectl scale deployment payment-api --replicas=5. The ASG activity log showed:

Launching a new EC2 instance. Status Reason: The security group 'sg-0deadbeef12345678' does not exist. Launching EC2 instance failed.

The launch template had been created six months earlier. Someone deleted the SG during a cleanup sprint without updating the template. The Binadox article on missing security group errors describes exactly this dormant failure condition Securing AWS Auto Scaling: Preventing Missing Security Group Errors.

The RDS Ingress Rule That Locked Out the New EKS Nodes

When the payment pods started, they failed PostgreSQL connection with connection timed out. The RDS SG ingress rule allowed 10.0.3.0/24. The EKS nodes ran in 10.0.4.0/24. A Terraform apply two weeks earlier had replaced 0.0.0.0/0 with the wrong subnet CIDR. The October 2024 RDS postmortem records the same mistake: a single CIDR replacement with the wrong VPC subnet caused 47 minutes of total outage Postmortem: How a Misconfigured Security Group Blocked Our RDS Access.

What We Tried First and Why It Failed

Attempt 1: Manual SG Edits in the Console During the Maintenance Window

At 02:14 IST we opened the console, navigated to sg-0a1b2c3d4e5f6g7h8, added Stripe's CIDR to egress. The webhook test still failed. Reason: the Istio sidecar in the payment namespace was set to REGISTRY_ONLY outbound traffic policy with no ServiceEntry for Stripe. The mesh dropped the traffic before it reached the SG. We wasted 18 minutes discovering this.

Attempt 2: Terraform Apply Without Pre-Plan Validation

We ran terraform apply -target=aws_security_group_rule.rds_ingress to fix the RDS CIDR. The plan showed the change. Apply succeeded. Pods still could not connect. The Terraform state had drifted: the SG rule in AWS was already 10.0.3.0/24 but the state file showed 10.0.4.0/24. A prior manual edit had caused drift. Terraform v1.9.0's aws_security_group_rule resource lacked mandatory CIDR validation until v1.10.2, as noted in the RDS postmortem.

Attempt 3: Relying on Istio ALLOW_ANY as a Temporary Bypass

We patched the Istio OutboundTrafficPolicy to ALLOW_ANY for the payment namespace. Webhooks started working. We declared victory and went to sleep. At 06:32 IST the on-call engineer got paged: the payment service was now making outbound calls to an internal admin panel that had no authentication. ALLOW_ANY bypassed all mesh-level controls. We reverted at 06:45 IST.

The Working Approach: Inventory, Rehearsal, Rollback

Step 1: Automated Dependency Mapping with aws ec2 describe-security-groups and jq

We now run this script before every cutover:

#!/usr/bin/env bash
set -euo pipefail

REGION="ap-south-1"
OUTPUT="sg-inventory-$(date +%Y%m%d).json"

aws ec2 describe-security-groups --region "$REGION" \
  | jq '[.SecurityGroups[] | {GroupId, GroupName, VpcId, Ingress: .IpPermissions, Egress: .IpPermissionsEgress, Tags}]' \
  > "$OUTPUT"

echo "Inventory written to $OUTPUT"

The JSON feeds a Python script that cross-references launch templates, ENIs, and RDS instances to produce a dependency graph. We commit the graph to the repo as sg-dependency-graph.json.

Step 2: Pre-Cutover Rehearsal in a Staging VPC Mirroring Production CIDRs

We spin a staging VPC with identical CIDR blocks (10.0.0.0/16, subnets 10.0.3.0/24, 10.0.4.0/24, 10.0.5.0/24). The rehearsal runs the full cutover playbook: DNS switch, ALB target group swap, RDS read-replica promotion, Stripe webhook endpoint update. We validate every external dependency, Stripe, SendGrid, S3, CloudWatch, from the staging EKS nodes. The rehearsal must pass without manual intervention.

Step 3: Rollback Runbook with aws ec2 revoke-security-group-egress and authorize-security-group-egress Commands

Every SG change during cutover is paired with a rollback command stored in rollback.sh:

#!/usr/bin/env bash
set -euo pipefail

# Rollback Stripe egress addition
SG_ID="sg-0a1b2c3d4e5f6g7h8"
STRIPE_CIDR="3.104.0.0/14"  # example only

aws ec2 revoke-security-group-egress \
  --group-id "$SG_ID" \
  --ip-permissions "IpProtocol=tcp,FromPort=443,ToPort=443,IpRanges=[{CidrIp=$STRIPE_CIDR}]"

# Rollback RDS ingress CIDR
RDS_SG="sg-0r1d2s3q4l5e6r7s8"
OLD_CIDR="10.0.3.0/24"
NEW_CIDR="10.0.4.0/24"

aws ec2 revoke-security-group-ingress \
  --group-id "$RDS_SG" \
  --ip-permissions "IpProtocol=tcp,FromPort=5432,ToPort=5432,IpRanges=[{CidrIp=$NEW_CIDR}]"

aws ec2 authorize-security-group-ingress \
  --group-id "$RDS_SG" \
  --ip-permissions "IpProtocol=tcp,FromPort=5432,ToPort=5432,IpRanges=[{CidrIp=$OLD_CIDR}]"

We test the rollback script in staging before the maintenance window. It must complete in under 90 seconds.

Step 4: CI Gate Using Checkov 2.4.0 Custom Rules for SG CIDR Validation

Our GitHub Actions workflow includes:

- name: Checkov SG validation
  uses: bridgecrewio/checkov-action@v12
  with:
    version: 2.4.0
    directory: ./infra
    framework: terraform
    check: CKV_AWS_188,CKV_AWS_189,CKV_AWS_190
    skip_check: CKV_AWS_20

We added custom rules in checkov/custom_rules/sg_cidr.yaml that reject any aws_security_group_rule with cidr_blocks containing 0.0.0.0/0 or CIDRs outside the VPC range defined in variables.tf. The RDS postmortem notes that adding three validation layers reduced SG misconfigurations by 94 percent in three months.

Pitfalls We Warn Every Intern About

Assuming Stateful Means Symmetric, Egress Rules Still Matter

Security groups are stateful for allowed inbound connections. Return traffic for an established connection flows regardless of egress rules. But new outbound connections, like a pod calling Stripe, require explicit egress allow. The DEV Community post by Tejas Shinkar illustrates this: restricting egress from the application SG killed database connectivity even though the DB SG allowed inbound from the app SG I broke my own AWS infrastructure… without touching the infrastructure.

Deleting an SG Before Updating Every Launch Template That References It

The ASG launch template failure was 100 percent preventable. AWS Config custom rules can validate that every launch template's security_group_ids exist. We now tag every SG with owner:team-platform and dependents:lt-0abcdef1234567890,eni-0123456789abcdef0. Deletion requires a PR that updates all dependents.

Using 0.0.0.0/0 in Rehearsal Then Forgetting to Tighten Before Cutover

During staging rehearsal we temporarily opened 0.0.0.0/0 on the RDS SG to unblock testing. The cutover runbook now includes a mandatory step: grep -r "0.0.0.0/0" infra/ && echo "FAIL: open CIDR found" && exit 1. No open CIDRs allowed in the cutover branch.

Trusting Terraform Plan Output Without Running checkov -d . First

Terraform plan shows what will change. It does not validate semantic correctness of CIDR values. The RDS postmortem shows Terraform v1.9.0 allowed an invalid subnet CIDR without warning. We run checkov -d ./infra --framework terraform as a required CI gate. Plan output is reviewed only after Checkov passes.

What We Would Do Differently Next Time

Codify the Full SG Dependency Graph in a Versioned JSON Manifest

The spreadsheet worked for one migration. For the next we will generate sg-dependency-graph.json from the inventory script and store it in the repo with a git tag cutover-2026-01-15. The manifest includes every SG, every rule, every resource that references it, and the external CIDRs required. Changes to the manifest require a PR with Checkov validation.

Add a Mandatory aws configservice put-config-rule for SG Drift Detection

We will deploy an AWS Config rule that runs every six hours:

aws configservice put-config-rule \
  --config-rule '{
    "ConfigRuleName": "sg-drift-detection",
    "Description": "Detect drift between Terraform state and live SG rules",
    "Scope": {"ComplianceResourceTypes": ["AWS::EC2::SecurityGroup"]},
    "Source": {
      "Owner": "CUSTOM_LAMBDA",
      "SourceIdentifier": "arn:aws:lambda:ap-south-1:123456789012:function:sg-drift-checker",
      "SourceDetails": [{"EventSource": "aws.config", "MessageType": "ConfigurationItemChangeNotification"}]
    }
  }'

The Lambda compares live SG rules against the committed sg-dependency-graph.json and posts findings to Slack.

Schedule a Dry-Run Cutover 72 Hours Before the Real Window

The Thursday rehearsal caught two of three issues. A Tuesday dry-run (72 hours before Saturday cutover) would have caught the third, the deleted SG reference, because the ASG scale-out test would have run earlier. We now mandate a dry-run at T-72h with the same runbook, same rollback script, same validation gates.

Replace Manual Rollback Steps with a Tested Lambda Function Triggered by CloudWatch Alarm

The rollback.sh script worked but required a human to run it. Next time we will package the rollback logic into a Lambda function invoked by a CloudWatch alarm on ELB 5XX count > 10 for 2 minutes. The Lambda will execute the revoke/authorize commands, post to Slack, and create an incident ticket. We will test the Lambda in staging during the dry-run.

---

The cutover succeeded on the second attempt, one week later. Zero downtime. The inventory spreadsheet, rehearsal VPC, rollback script, and Checkov gate are now standard in our migration playbook. Every intern walks through them before touching a production VPC.