CS Fundamentals Trainer · Guides

Load balancing

Layer 4 versus layer 7, a simulation of random, round robin, least connections and power of two choices, health checks, draining and sticky sessions.

12 min read

A load balancer sits in front of several identical servers and spreads incoming requests across them. It gives you three things at once: more capacity than one machine, availability (a dead server stops receiving traffic), and freedom to deploy (take servers out one by one). The interesting part is the choice of which server gets each request, and how the balancer knows which servers are healthy. This guide includes a simulation, so the differences between policies come with numbers instead of adjectives.

1. Where it sits and what it does

Diagram: A load balancer in front of three servers, with health checks

Its jobs: pick a backend for each request or connection, stop sending traffic to unhealthy backends, and often also terminate TLS, compress, rate limit, and add headers (see the TLS guide). The balancer itself must not be a single point of failure, so in practice there are two or more, sharing one address or sitting behind DNS (section 8).

2. Layer 4 or layer 7

Layer 4 (transport)Layer 7 (application)
Decides usingIP addresses and portsThe HTTP request: host, path, headers, cookies
Sees the contentnoyes
Costvery low (forward packets or connections)higher (parse requests, often terminate TLS)
Can route /api and /images differentlynoyes
Can retry a failed request on another serveronly at connection timeyes, per request
ExamplesAWS Network Load Balancer, HAProxy in TCP mode, IPVSNGINX, Envoy, HAProxy in HTTP mode, AWS Application Load Balancer

A common arrangement uses both: a layer 4 balancer spreads connections across a pool of layer 7 proxies, which route to services. One detail matters for layer 7 and HTTP/2: a client keeps one long connection and multiplexes many requests over it, so a layer 4 balancer that balances connections can send all of one busy client's requests to one server. For gRPC and HTTP/2 traffic balance per request at layer 7.

3. Choosing the backend

Here are the four policies you should know, then a simulation that compares them.

Two more are used for a different reason:

What the simulation shows

I wrote a small simulation: 10 servers, each handling one request at a time with a first-come queue; requests arrive at random so that the servers are 90% busy on average; 200,000 requests. Times are in units of the average service time (1.0). I ran it with three kinds of request, all with the same mean of 1.0: exactly equal sizes; exponentially distributed sizes (the usual textbook model: many short requests, a few long ones); and heavy-tailed (90% short requests, 10% about eleven times longer).

PolicyEqual sizes: mean / p99Exponential sizes: mean / p99Heavy-tailed: mean / p99
Random5.7 / 23.110.3 / 45.125.4 / 118.7
Round robin1.4 / 3.25.3 / 23.727.3 / 133.1
Least connections1.4 / 3.01.9 / 7.73.6 / 27.2
Power of two choices1.9 / 4.23.0 / 10.36.3 / 34.9

(Medians over five runs with different random seeds, in units of the mean request time. The simulation is a model, not a benchmark of any real balancer: every server is identical, there is no network delay, and the load is high on purpose so the differences are visible.)

What to take from it:

4. Health checks

A balancer must stop sending requests to a server that is down, and start again when it recovers.

What the check endpoint should test is a design decision:

Other behaviour that goes with it:

5. Sessions and sticky routing

If each server keeps a user's session in its own memory, the second request must go to the same server ("sticky sessions", done by a cookie or by hashing the client). The cost: load is less even, and when that server dies, the sessions are gone.

The usual better answer is to keep servers stateless: put the session in a shared store (Redis, a database) or in a signed token the client holds, so any server can handle any request. Then the balancer can use any policy and servers can come and go freely.

6. Example configuration

The same idea in two real tools (the exact option names are in each tool's documentation [3][4]).

# NGINX: least connections, with passive health checking and a backup server
upstream app {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.13:8080 backup;
}
server {
    listen 80;
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_502 http_503;   # retries; only safe for idempotent requests
        proxy_next_upstream_tries 2;
    }
}
# HAProxy: HTTP mode, active checks, two choices
backend app
    balance random(2)            # power of two choices
    option httpchk GET /healthz
    default-server inter 5s fall 2 rise 3 slowstart 30s
    server s1 10.0.0.11:8080 check
    server s2 10.0.0.12:8080 check
    server s3 10.0.0.13:8080 check

7. The policy in code

Selecting a server is a few lines. Here is power of two choices with a count of in-flight requests per server (pick a language on the tabs; your choice is remembered):

import random

class Pool:
    def __init__(self, servers):
        self.servers = list(servers)
        self.inflight = {s: 0 for s in self.servers}

    def pick(self):
        if len(self.servers) == 1:
            return self.servers[0]
        a, b = random.sample(self.servers, 2)               # two distinct servers
        return a if self.inflight[a] <= self.inflight[b] else b

    def start(self):
        s = self.pick()
        self.inflight[s] += 1
        return s

    def finish(self, s):
        self.inflight[s] -= 1
type Pool struct {
	mu       sync.Mutex
	servers  []string
	inflight map[string]int
}

func (p *Pool) Start() string {
	p.mu.Lock()
	defer p.mu.Unlock()
	a := p.servers[rand.IntN(len(p.servers))]
	b := p.servers[rand.IntN(len(p.servers))] // may equal a; with many servers that is rare
	s := a
	if p.inflight[b] < p.inflight[a] {
		s = b
	}
	p.inflight[s]++
	return s
}

func (p *Pool) Finish(s string) {
	p.mu.Lock()
	p.inflight[s]--
	p.mu.Unlock()
}
class Pool {
  constructor(servers) {
    this.servers = [...servers];
    this.inflight = new Map(this.servers.map((s) => [s, 0]));
  }
  start() {
    const a = this.servers[Math.floor(Math.random() * this.servers.length)];
    const b = this.servers[Math.floor(Math.random() * this.servers.length)];
    const s = this.inflight.get(b) < this.inflight.get(a) ? b : a;
    this.inflight.set(s, this.inflight.get(s) + 1);
    return s;
  }
  finish(s) {
    this.inflight.set(s, this.inflight.get(s) - 1);
  }
}
public class Pool
{
    private readonly string[] _servers;
    private readonly Dictionary<string, int> _inflight;
    private readonly object _lock = new();

    public Pool(IEnumerable<string> servers)
    {
        _servers = servers.ToArray();
        _inflight = _servers.ToDictionary(s => s, _ => 0);
    }

    public string Start()
    {
        lock (_lock)
        {
            var a = _servers[Random.Shared.Next(_servers.Length)];
            var b = _servers[Random.Shared.Next(_servers.Length)];
            var s = _inflight[b] < _inflight[a] ? b : a;
            _inflight[s]++;
            return s;
        }
    }

    public void Finish(string s)
    {
        lock (_lock) { _inflight[s]--; }
    }
}

Always call finish in a finally block, or a failed request leaves the count too high forever.

8. Making the balancer itself highly available

9. Common mistakes

10. Things to mention in an interview

Quick recap

  1. A load balancer spreads requests over servers and removes dead ones.
  2. Layer 4 is fast and blind; layer 7 can route on the content and retry per request.
  3. Random and round robin ignore load; least connections and power of two choices look at it and do much better (see the table).
  4. Health checks should be cheap and should separate "restart me" from "do not send me traffic".
  5. Retry only idempotent requests and only a few times.
  6. Stateless servers make everything easier than sticky sessions.
  7. Make the balancer redundant too.

Sources

  1. Yossi Azar, Andrei Broder, Anna Karlin and Eli Upfal, Balanced Allocations, SIAM Journal on Computing, 1999. The balls-into-bins result behind two random choices.
  2. Michael Mitzenmacher, The Power of Two Choices in Randomized Load Balancing, IEEE Transactions on Parallel and Distributed Systems, 2001. The same idea for servers with queues.
  3. NGINX documentation, HTTP load balancing and ngx_http_upstream_module.
  4. HAProxy documentation, Configuration manual (the balance keyword, option httpchk, and server check, inter, fall, rise, slowstart).
  5. Envoy documentation, Load balancing and the least request policy settings.
  6. IETF, RFC 9110: HTTP Semantics. Which methods are idempotent and so safe to retry.

Practise this topic: Resilience & Scalability Load balancing Networking

More guides