Load Balancers

It is the IPL final, and the innings break has just begun. Orders on a food delivery app jump fivefold in two minutes. One of its twelve API servers runs out of memory and stops answering, but the load balancer in front still thinks it is healthy: it checks each server every 30 seconds and waits for 3 misses in a row. For the next minute and a half, every twelfth order spins and fails, and hungry customers tap "Place order" again and again. Tuned checks would have pulled it out within seconds.

A single server runs out of something eventually: CPU, memory, bandwidth or open connections. More servers fix that but create three new problems. Clients need one stable address, traffic must be spread across the servers, and a dead server must stop receiving requests.

A load balancer solves all three, much like the host at a busy restaurant who seats each arriving group at a free table and stops seating anyone at a broken one. It accepts traffic on one address and forwards each connection or request to one server in a pool behind it, checking the servers' health as it goes. It can be software, an appliance or a managed cloud service. For the food app it buys five things:

• Scale: add servers to the pool as orders grow, and customers never notice.
• Availability: a failed server leaves the rotation automatically, within seconds when the checks are tuned.
• Deploys without downtime: servers restart one at a time while the rest keep serving.
• A smaller attack surface: only the balancer faces the internet.
• One place for shared work: encryption, compression, request logs and basic limits.

Availability, Reliability, Scalability and Latency showed that three servers survive a failure better than one. The load balancer is what turns a pool of machines into one service that keeps answering when a machine dies.

The front door is not the only place a balancer appears. Wherever one tier calls a pool of identical machines, something must choose a machine. A typical order on the food app passes through two balancers:

Read replicas of a database follow the same idea, often balanced by the database driver itself. Choosing between whole regions is a global job, covered later in this chapter.

Between internal services, the caller often does the balancing itself. With client-side load balancing, each service fetches the list of healthy instances from a service registry, one of the discovery options in DNS: How a Name Becomes an Address, and picks an instance per request, with no box in the middle. A service mesh does the same through a small proxy beside each instance. Both save a hop and a box that could fail, at the cost of a library or proxy in every caller.

Before the algorithms, one naming question: is the box in front a load balancer, a reverse proxy or an API gateway?

Three names for the box in front of your servers are often used as if they meant the same thing. They overlap, and one product can play all three roles, but each name puts a different job first:

The lines blur: an L7 load balancer is a reverse proxy that balances, and an API gateway is a reverse proxy with API rules added.

A common layout puts a managed balancer at the edge, an API gateway behind it, and internal balancing in front of each service. On the food app, the gateway rejects calls without a valid login token and caps each restaurant partner's API key at a set rate; Rate Limiters covers how those limits work.

The first real choice is how much of each request the box reads. Balancers are named after the layer of the network model they work at. A Layer 4 (L4) balancer works with TCP and UDP: it sees addresses and ports, picks a server when a connection opens, and passes the bytes through unread, like a sorter who routes letters by the envelope alone. A Layer 7 (L7) balancer speaks the application protocol, usually HTTP, and opens the envelope: it reads each request's path, headers and cookies before choosing. The table compares them:

L4 is the cheap, fast pipe; L7 spends more CPU per request to route by content.

That routing lets one address front many services. The food app sends /api/orders to the order service, /api/menu to the menu service and /ws to its live order-tracking servers, all behind api.example.in.

Once the balancer has a pool, it needs a rule for picking a server and a way to notice a dead one. The simplest rule is round robin: send each new request to the next server in the list, and wrap around at the end. It needs no knowledge of the servers and suits identical servers with similar requests, like most of the food app's API calls.

Round robin alone would keep feeding a dead server. Active health checks fix that: the balancer calls a cheap endpoint such as GET /health on every server at a set interval, waits up to a timeout for the reply, removes a server after a number of failures in a row (the unhealthy threshold) and returns it after a number of successes (the healthy threshold). An L4 check may only open a connection; an HTTP check expects 200 OK.

Request 4 failed although the balancer was checking, because a server leaves the pool only at the unhealthy threshold, and every request sent before then is lost. The length of that gap is a setting you choose, and the next page does the arithmetic.

Back to the IPL final. The food app's balancer checked every 30 seconds and needed 3 misses, so how long did the dead server keep receiving orders? The gap is roughly the interval times the unhealthy threshold, plus the last check's timeout:

detection ≈ interval × unhealthy threshold + timeout
the story:  30 s × 3 + 5 s ≈ 95 s
tuned:       5 s × 2 + 2 s ≈ 12 s

Worst case, when the crash lands just after a passed check. Requests lost ≈ detection time × that server's request rate.

At the dinner peak each of the 12 API servers handled about 1,000 requests a second, so the story's 95 seconds cost up to 95,000 failed or hung requests. The tuned setting cuts that to about 12,000.

Why not check every second? Tighter checks fail in their own way. A healthy server that pauses for 3 seconds, say for a garbage collection, misses two quick checks and is thrown out. When many servers pause under the same load, the pool flaps, losing capacity exactly when it is busiest. The healthy threshold is the matching brake on re-entry: requiring 2 or 3 successes stops a server that crashes on every restart from rejoining and failing again.

Passive health checks close most of the remaining gap. The balancer watches real traffic, and when requests to one server keep failing, for example 5 errors or timeouts in a row, it removes the server without waiting for a probe, usually within a second or two at this traffic. Passive checks see only servers that get traffic, so setups use both.

Round robin assumes requests are alike. The food app's restaurant dashboard breaks that: most requests are quick menu edits, but a few are sales-report exports that run for 30 seconds. Round robin hands out work evenly by count, so one unlucky server can collect several exports and fall behind. Least connections sends each new request to the server with the fewest requests in progress, like joining the shortest queue at a ticket counter.

Work in progress is a cheap stand-in for load: a server stuck on slow work keeps its requests open, so it gets less new work, and the balancer needs no knowledge of CPU. An L4 balancer counts open connections; an L7 balancer counts requests in flight, which stays accurate when one connection carries many requests. One catch: a brand-new server shows zero and gets flooded, so balancers ramp its share up gradually, a slow start.

Round robin and least connections are two of seven common algorithms. Pick the simplest one that fits your traffic, and change it only when you can name the problem it causes:

Most pools run round robin or least connections; the hashing rows matter only when a server holds something local, such as a warm cache.

The hashing rows raise a bigger question: should one user always reach the same server? Some apps keep a user's session, such as their login or the food app's cart, in the memory of the server that created it, so every later request must return there. Sticky sessions (session affinity) make the balancer do this, usually with a cookie it sets or with IP hashing. The cost shows up when anything changes:

Each problem comes from the same cause: the session lives on one machine.

The better fix is to make servers stateless, as Availability, Reliability, Scalability and Latency described: keep sessions in a shared store such as a distributed cache, read in under a millisecond, or in a signed token (for example a JSON Web Token, or JWT, in a cookie) that the client sends with every request. Any server can then handle any request, and the balancer is free to use whichever algorithm suits the traffic.

Affinity is still fine when losing it costs only speed: a user routed to the server whose local cache holds their data is just slower elsewhere if it dies. WebSockets are naturally pinned, since one connection lives on one server.

Every request from the food app arrives encrypted, so some machine must decrypt it. Which one? TLS (Transport Layer Security, the encryption behind HTTPS) costs one round trip on top of the TCP handshake with TLS 1.3, or two with the older TLS 1.2, plus public-key maths on the server. Terminating TLS at the load balancer means the balancer holds the certificate and decrypts there. An L7 balancer must decrypt anyway to read paths and headers, and the handshake maths then runs on the balancer instead of every app server. There are three options:

The food app terminates at the balancer for menus and tracking, and re-encrypts the payment path, because card and UPI details should never cross even a private network in plain text.

Terminating at the balancer also gives one place to renew certificates and resume sessions, so returning clients skip part of the handshake. HTTP/3 runs over QUIC, on UDP, and merges the transport and TLS handshakes into one round trip; balancers usually end it at the edge and speak HTTP/1.1 or HTTP/2 to the servers behind them.

Everything so far happens inside one region. When the food app runs in Mumbai and Hyderabad, something must also decide which region each customer reaches. A local load balancer spreads traffic across servers in one region; a global load balancer picks the region: the nearest for speed, and a healthy one when a region fails. Two techniques do the global job:

Most teams start with DNS-based balancing and accept its minutes of failover, because anycast needs a large provider's network.

So the layers stack: a customer in Kolkata looks up api.example.in, the global layer picks Mumbai, and Mumbai's local balancer picks a healthy server. If Mumbai fails, the global layer moves new lookups to Hyderabad, while customers holding a cached answer keep trying Mumbai until it expires, as DNS: How a Name Becomes an Address showed. The global layer fails over in minutes; the local layer in seconds.

Put one load balancer in front of the food app's twelve servers and you have built a new single point of failure: if that box dies, all twelve are unreachable. The standard fix is a pair of balancers sharing a virtual IP (VIP), an address that can move from one machine to another:

A pair can also share the work instead of keeping one idle. The two arrangements compare like this:

Either way, the spare capacity is the price of surviving a failure.

VRRP (Virtual Router Redundancy Protocol) is the standard protocol for this takeover, and very large sites put a layer of L4 balancers in front of a fleet of L7 ones. Managed cloud balancers are already fleets across zones; you still spread them, and the servers behind them, over more than one zone.

A redundant balancer protects you from its own failure. Deploys are the failure you cause yourself: a new release of the food app restarts 12 servers one at a time, and restarting a server mid-request fails those requests. Connection draining lets a server leave gracefully: no new requests, and time for the current ones to finish.

The order of the steps matters, because balancers learn slowly that a server is leaving: a check must fail, or a deregistration call must reach every balancer node. The full shutdown sequence:

The step teams skip is the third. A server that stops accepting connections the moment it gets the stop signal is still on the balancers' lists for several seconds, so they keep sending it requests, which are refused, and customers see 502 Bad Gateway errors on every deploy. Keep serving until the balancers have stopped sending, then stop.

Two kinds of check keep this clean. A readiness check answers "may I get traffic?" and a liveness check answers "should I be restarted?"; draining fails only the first. A liveness check must never test shared dependencies, or a database blip restarts every server at once.

Draining works when requests finish in seconds. The food app's live order tracking does not: each WebSocket stays open for the whole order, on the one server it first reached. One connection server can hold about 100,000 mostly idle sockets, so restarting it disconnects 100,000 customers at the same moment.

They all reconnect at once. If they return within 10 seconds, that is 10,000 new connections a second, each with a TCP and TLS handshake, a login check and a sync of missed rider updates. This reconnect storm hits the balancer, the login service and the remaining servers, once per server in a rolling deploy, and leaves each restarted server empty while the others hold its old clients. Four habits tame it:

• Drain slowly. Close a server's sockets in batches over a few minutes, or ask clients to reconnect after a random delay.
• Back off with jitter. Each client waits a random, growing delay before retrying, such as up to 1 s, then 2 s, then 4 s, so retries spread out instead of arriving in waves.
• Balance new sockets by least connections, so reconnecting clients fill the emptiest servers, including the one just restarted.
• Keep headroom for the burst. Size the login service for the reconnect rate, and let returning clients resume their TLS session to skip part of the handshake.

One more tool covers the gaps that checks and draining leave: the balancer can retry a failed request on another server. It hides failures well, and it can also cause harm.

A retry is safe when the first attempt never reached a server, such as a refused connection, or when the request is safe to repeat, such as GET /api/menu. It is not safe for POST /orders: the first server may have placed the order just before timing out, so a retry places it twice. That is the double-action problem Abstractions That Hide the Network solved with idempotency keys. Let the balancer retry only safe methods, or requests that carry an idempotency key.

Retries also multiply load. During an overload every failed request becomes two, so a struggling pool gets more work just when it can least cope: the retry storm from Availability, Reliability, Scalability and Latency. Retry at most once, within a budget such as 10% extra requests, and fail fast beyond it.

The balancer sees every request, which makes it the best place to watch the whole pool:

Draw a load balancer wherever a pool of identical servers takes traffic, and say which layer, which algorithm and what happens when something fails.