Twenty-Seven Knocks: How Our Own Retry Logic DDoSed the Payments Service

Nobody attacked us. We did it to ourselves, three polite retries at a time.

Act I: The Slowdown

At 14:07 on a Tuesday, PagerDuty lit up my phone: payments-service p99 > 2000ms. Checkout error rate: 11% and climbing.

Our checkout path was a textbook microservices chain. A Node.js API gateway called a Python order-service, which called a Go payments-service, which wrote to Postgres. Different teams built every hop over three years, and each did the "responsible" thing: wrap the downstream call in a retry.

What started it was mundane. A nightly reconciliation job ran late and grabbed row locks on the payment_intents table during lunch traffic. Charges that normally took 180ms started taking 1.5–2 seconds.

A slow dependency can ruin an afternoon. What we built on top of it turned it into an outage.

Act II: The Rabbit Hole

False start #1: “It’s loaded. Scale it.”

Requests per second on /charge were up 6x, so we assumed a traffic spike and scaled payments-service from 6 pods to 18. Within four minutes, the logs changed from timeouts to this:

FATAL: sorry, too many clients already
pq: remaining connection slots are reserved for non-replication superuser connections