Load balancers, through one tap in a big sale: round robin, health checks, the server that dies mid-sale, and the cart that vanishes
It's 8 PM. The biggest sale of the year just opened, and you and a million other people tap Buy in the same minute.
Behind that one button there are fifty servers. Something has to decide which one gets your tap. And one of those servers is about to die, with an order still on it.
This article follows one tap through five steps: Choose → Balance → Detect → Survive → Scale. If you'd rather watch, here's the 9-minute animated version:
We'll follow Ananya as she shops the sale on a store at shop.example.com. The numbers are an illustrative model, not a benchmark.
Why one server isn't enough
One tap can trigger a few requests behind the scenes, but let's keep it simple: one request per tap. A million taps in a minute is about 16,667 requests every second.
In our example, one server handles about 500 a second. Ask it for 17,000 and it doesn't slow down politely. It falls over.
You could buy a bigger machine. That's scaling up. It hits a ceiling fast, and when that one machine dies, everything dies.
So you run many ordinary machines side by side. That's scaling out. 16,667 ÷ 500 ≈ 34 servers. You run 50, for spare room.
Which creates a new problem: with fifty servers, how does your phone know which one to talk to?
Step 1, Choose: the load balancer
It doesn't. Your phone knows one address, and that address belongs to a load balancer.
Every request arrives at the load balancer first. It picks a healthy server from its list, forwards the request, and the reply comes back to the phone. To Ananya it looks like one service. Behind it, fifty servers share the work, about 333 requests each, every second.
So how does it pick?
Step 2, Balance: round robin
The simplest rule is round robin: server 1, then 2, then 3, all the way to 50, then back to 1. When requests take roughly the same time, that works well.
But one shopper is buying a phone case, and another has forty items, three coupons and gift wrap. That request takes thirty times longer. Round robin hands out turns, not work, so server 9 is stuck on the heavy order while new requests keep landing in its queue, and servers 8 and 10 sit nearly idle.
Equal turns aren't equal work.
Least connections and weights
Least connections asks a better question: which server has the fewest active connections right now? The next request goes there, and server 9 gets a break until it catches up.
It doesn't measure CPU. Open connections are a rough signal of how busy a server is, and a cheap one to track.
If some machines are bigger, give them weights. A server with weight 2 can receive twice the share of a server with weight 1.
And some apps add one more rule: send the same user back to the same server every time. That's a sticky session. Remember it. It's going to hurt.
Step 3, Detect: health checks
A load balancer doesn't automatically know whether a server is alive, so it checks. Some checks just try to open a connection; others call a health endpoint in the app that says whether the service is ready.
In our example it checks every 2 seconds, and three misses in a row mark a server unhealthy. Real systems tune these numbers, but here that's about 6 seconds to notice.
At 8:00:04 PM, server 7 stops answering.
Step 4, Survive: the six seconds
For those six seconds, the balancer still thinks server 7 is fine. At about 333 requests a second, roughly 2,000 requests may be sent its way before it's removed.
Ananya's payment is one of them. She taps Pay. The spinner turns, and turns. Then: something went wrong.
Did she pay or not? Here's what actually happened. Server 7 saved her order to the database, then died before it could reply. The order was saved; the reply was lost.
So her app tries again, and the balancer sends the retry to server 23, which is alive. If nothing stops it, that's a second order. Charged twice.
Payment systems prevent this with an idempotency key: one unique ID for this one payment, sent with every retry. The key and its result are stored reliably in a place every server can reach. Server 23 looks up the key, sees the payment is already done, and returns the original result instead of charging her again.
POST /payments
Idempotency-Key: A7F3
→ first time: process the payment, store (A7F3 → result)
→ any retry: find A7F3, return the stored result
Pause and guess
Remember the sticky session? Ananya's cart was kept in server 7's memory: three headphones, a charger and a coupon.
Server 7 is gone. She refreshes the page. What does she see?
Her cart as it was? An empty cart? An error? Make your guess before you read on.
The cart that vanished
An empty cart. The balancer sent her to a healthy server, but that server had never heard of her. Her cart died with server 7.
That's the trap. When something a user needs lives inside one server's memory, that server can't be replaced. That something is called state.
The fix is to take the state out of the servers. Keep carts and sessions in a shared session store, such as a fast cache like Redis, set up so the data survives the failures your app needs to handle. Every server can read it, so the servers become stateless: any healthy server can handle any request.
Ananya refreshes again, lands on server 41, and her cart is right there.
Layer 4 vs Layer 7
Some load balancers work like a postman who only reads the envelope. A Layer 4 load balancer looks at the IP address, the port and the transport protocol, and forwards the packet without opening it. Very fast, very simple.
A Layer 7 load balancer opens the envelope. It reads the request itself: the path, the headers, even which website name was asked for. So /checkout goes to the checkout servers and /images goes to the image servers.
| Layer 4 | Layer 7 | |
|---|---|---|
| Reads | IP, port, protocol | Path, headers, host |
| Speed | Very fast | More work per request |
| Can route by | Connection | What the request asks for |
Many big systems use both: a fast Layer 4 balancer in front, and Layer 7 routing behind it.
Step 5, Scale: who balances the balancer?
If every request goes through one load balancer, that balancer is the new single point of failure. So you run more than one.
Active-passive: one handles the traffic while a standby waits, ready to take over the address if the first one fails.
Active-active: they share the traffic, and if one fails the others carry on, provided they have spare capacity.
Either way, failover is automatic, usually within seconds, depending on how it's set up.
For users all over the world there's one more layer. DNS-based global traffic management directs users towards a suitable region based on location, availability and routing policies. Ananya might be sent to Mumbai while someone in Jakarta goes to Singapore, and each region has its own load balancers and servers. That's a different job from a CDN, which serves cached files like images from the edge.
Every piece on one screen
DNS sends you to a suitable region.
A pair of load balancers share one address, so either can fail.
They pick a healthy server by turns, by fewest connections, or by weight.
Health checks take dead servers out within seconds.
An idempotency key makes a retry return the original result instead of a second charge.
A shared session store keeps the cart, so any server can pick up where another left off.
One address on the outside. Fifty servers on the inside. And even when one fails, the service keeps going.
Watch the full animated version: Click here
What was your guess for Ananya's cart? And has a retry ever charged you twice? 👇
Before her tap even reaches the load balancer, it crosses DNS, routers, TCP and TLS. That part of the trip is in the Networking video: https://youtu.be/sFySVuvONWA
Follow The Byte Mechanics
How software works, under the hood ⚙️
Top comments (0)