Load balancing

The big picture and the words we use

Let us start with a few words you will see all the time. A client is whoever asks for something (your browser, your phone app). A request is one question the client sends, like "give me the home page". A server answers. When we put many equal servers behind one door, each of them is called a back end (or an instance, a copy of the same program). The door is the load balancer.

            ┌──────────┐      ┌─> back end 1
clients ──> │   load   │ ─────┼─> back end 2
            │ balancer │      └─> back end 3
            └──────────┘
  one address         many equal copies
  the clients see     of the application

The clients only know the address of the balancer. They never learn how many back ends exist. That is very useful: we can add or remove back ends and nobody outside notices.

The two jobs of a load balancer

• Share the work: so that no single server gets crushed while others sit idle. This gives more capacity (more requests per second).
• Survive failures: when a server dies, stop sending requests to it. The users keep being served by the others. This gives availability.

Horizontal against vertical scaling

Your shop gets more customers than it can serve. There are two things you can do. You can buy a giant, super-fast machine to replace the old one. Or you can buy more ordinary machines and use them together. These two roads have names.

Why stateless services scale out easily

A service is stateless when any copy can answer any request. The data that must survive (sessions, carts, orders) lives in shared storage, like a database. Then the balancer can send each request anywhere, and copies can appear or disappear freely.

Sizing: N+1

If one instance handles 400 requests per second and the busiest moment brings 1500, you need ceil(1500 / 400) = ceil(3.75) = 4 instances just to serve the peak (we always round up). Call this number N. To still serve the whole peak when one instance dies, you add one spare: N + 1 = 5.

The spare is cheaper when machines are small. To survive one failure with 2 big machines, each able to carry 100% of the peak, you pay 200% of the peak. With 8 small machines, each carrying 1/7 of the peak, you pay 8/7, about 114%. One spare in 8 is a smaller fraction than one spare in 2.

Elasticity and its traps

With horizontal scaling you can switch instances on at the busy hour and off in the quiet hours, one at a time, while the others keep working. This is called elasticity. A vertically scaled machine stays sized for the peak all day, or has to be stopped at every resize.

Reverse proxy, load balancer and API gateway

A proxy stands between two parties. The question is: whom does it work for?

A reverse proxy is a natural place to do jobs that every request needs: TLS (the HTTPS padlock), caching, compression, hiding the real address of the origin and holding slow clients so the application is free for the next request. So a reverse proxy is useful even with a SINGLE server behind it. What it cannot give is redundancy, which needs more than one server.

A layer 7 load balancer is a reverse proxy that also decides which of several equivalent copies serves each request. That is why NGINX and Caddy play both roles with the same configuration.

An API gateway is a reverse proxy specialised in APIs. At one single point it applies rules common to all routes: who may call (authentication), how often (rate limiting), which service each route goes to and how requests or responses are adapted. The business logic stays in the services.

                        ┌─> orders replica 1
               /orders  │   (balancer of orders)
 clients ─> API ────────┼─> orders replica 2
           gateway      │
               /users   ├─> users replica 1
                        │   (balancer of users)
                        └─> users replica 2

What changes when a proxy is in the middle

The proxy keeps two separate connections: one with the client, one with the back end. This brings three practical consequences.

• The application sees the proxy's address, not the client's. The proxy passes the client address in a header called X-Forwarded-For. It is a list that grows to the right: each proxy appends the address of whoever handed it the request. With honest hops, X-Forwarded-For: 203.0.113.7, 10.0.0.5, 10.0.0.9 means the client is 203.0.113.7 (the first), then came proxy 10.0.0.5, and 10.0.0.9 is another hop. The last proxy does not appear in the list, because it is the peer of the connection the application sees.
• Never trust that header blindly. Any client can send X-Forwarded-For with a made-up value, and proxies append at the END. So the leftmost entry is the one the client controls. For rate limits or access control, read from the right, skip the addresses of your own trusted proxies, and take the first one left.
• Hop-by-hop headers are removed. Some headers (Connection, Keep-Alive, Transfer-Encoding, Upgrade) are valid only for one connection between two neighbours, as opposed to end-to-end headers that travel from the client to the origin. Since the proxy has two independent connections, it drops the hop-by-hop headers it received and creates the ones that fit the other connection.

Layer 4 against layer 7

Networks are described as a building with floors, called layers. The OSI model has seven: physical, data link, network, transport, session, presentation and application. Only two matter here. Layer 4 is the transport floor, where TCP and UDP live, with addresses and ports. Layer 7 is the application floor, where HTTP lives, with URLs, headers and cookies. "L4" and "L7" tell you up to which floor a balancer reads the traffic before deciding.

So the rule for choosing is simple: look at what your routing rule needs. Separating services by different addresses or ports works at layer 4. If the difference is in the URL path (/api/* here, /static/* there, same host and same port), in a header or in a cookie, that information only appears after reading HTTP, so you need at least layer 7.

The long connection trap

A layer 4 balancer chooses once per connection. If clients keep one connection open and send thousands of requests through it (HTTP/1.1 keep-alive), they all go to the same back end, and connections with different volumes make uneven load. The same happens, more strongly, with HTTP/2 and gRPC, which multiplex many calls over one connection. One very busy client can overload one back end while two others sit idle. The fix is to balance at the right unit, the request or the stream: a layer 7 proxy, or a client that opens connections to several back ends and alternates.

layer 4:  [conn A: req req req req req ...] -> back end 1
          [conn B: req ................] -> back end 2

layer 7:  req -> back end 1
          req -> back end 2
          req -> back end 3   (one choice per request)

With HTTPS, a layer 4 balancer can work in passthrough: it forwards the encrypted bytes and the secret tunnel goes from the client to the back end. It still sees what is outside the encryption (addresses, ports, when connections open and close), but path, headers and cookies are invisible to it.

Balancing algorithms

An algorithm for balancing answers one question: which back end gets the next request? Let us meet the most common ones.

Round robin

The back ends receive requests in turns, one after the other, and when the list ends it starts again, like dealing cards around a table. It ignores both the load and the speed of each server. With back ends A, B, C starting at A, request number k goes to position (k − 1) mod 3 of the list, counted from zero. For k = 7, (7 − 1) mod 3 = 0, so the 7th request goes to A.

Weighted round robin

Each back end has a weight, a number written in the configuration. The share of a server is its weight divided by the sum of all weights. With weights 3, 2 and 1, the sum is 6, so the shares are 3/6, 2/6 and 1/6. Of 600 requests: 300, 200 and 100.

The weights are static: they are a guess made in advance and do not change by themselves when a server becomes slower.

Least connections

The balancer keeps a counter for each back end: it adds 1 when it sends a request and subtracts 1 when the request ends. The next request goes to the back end with the fewest requests in progress. It is a simple estimate of how busy each server is. It clearly beats round robin when the requests have very different durations, because round robin equalises the number of requests sent, not the work.

Weighted least connections divides the active connections by the weight and picks the smallest result. Example: A has 12 connections and weight 1, B has 20 and weight 4, C has 9 and weight 1. The ratios are 12, 5 and 9, so B is chosen: a weight-4 server can hold four times more.

A slow server, two algorithms

Take three back ends: two answer in 10 ms and one in 40 ms, and a fixed number of clients keep sending the next request as soon as the last one ends. With round robin, each back end gets one third of the requests. But the slow one needs four times longer per request, so requests pile up on it. With least connections, every back end holds the same number of requests in progress. A back end that holds a number of requests finishes them at a rate of (in progress) divided by (time per request), so the rates are in the proportion 1/40 : 1/10 : 1/10 = 1 : 4 : 4. These add up to 9 parts, so the slow back end gets 1/9 of the requests, about 11%, and each fast one 4/9, about 44%.

Random and the power of two choices

Random picks a server by chance for each request. Over a few requests the split looks uneven, but over many each of N equal servers gets about 1/N. It needs no counter and no position, so several balancers can apply it without talking to each other. The power of two choices (random two in NGINX, random_choose 2 in Caddy) draws two servers by chance and keeps the less loaded of the two. This is far more balanced than plain random and costs only one comparison. It also reduces the "herd effect" in which several balancers all send everything to the one server that looks freest.

Hashing: IP hash and URL hash

A hash of a key picks the server. With IP hash the key is the client address, so the same client returns to the same server without any table (this holds while the set of servers does not change). With a URL hash the key is the requested address, so the same URL always goes to the same server. This is the best fit for caches: each node keeps responses for the URLs it serves, and the hit rate rises.

Two more traps with least connections

• Not shared counters. If two balancer processes keep separate counters, each decides with half of the picture and their decisions can pile on the same back end. The state must be shared (a shared memory zone), or you use a policy that tolerates partial information.
• The new empty server. A fresh server has zero connections, so in a busy pool it attracts almost all the new traffic at once, while it is still cold (empty caches, no open connections). A slow start raises its share gradually.

Sticky sessions (session affinity)

Sometimes a user must come back to the same back end. Session affinity (also called a sticky session) is the rule by which the balancer sends all the requests of one client to the same back end, instead of choosing again every time. The balancer recognises the client by a cookie or by the IP address.

Why would anyone need it? Because of state kept inside an instance. If your login or shopping cart lives only in the memory of the process that served you, a request sent to another instance finds nothing. Pinning the client to the instance works around that problem but does not solve it.

Two ways to recognise the client

What affinity does not give you

• It does not protect the state. If the session exists only in the memory of one instance and that instance crashes, the session dies. Failover only makes sure the user reaches a server that does not know them.
• It hurts autoscaling. The overloaded instances keep every session they already had, and the new instances receive only clients that are not bound yet. The longer the sessions, the slower the redistribution. Stateless services rebalance on the very next request.
• It can unbalance the load when addresses are shared (see IP hash).

The real cure: remove the state

Affinity is needed only while session state lives inside an instance. Move the state out, either to a shared store that every instance queries or to a signed token that the client sends with every request, and the instances become stateless. The balancer can choose any of them, and instances can crash or be added without losing sessions.

Consistent hashing

Hashing a key to choose a server has one big problem. The simple recipe is hash(key) mod N, where N is the number of servers (mod means the remainder after division). If N changes, the answer for almost every key changes.

Check it with numbers. Keys 0, 1, 2, ... with key mod N. Going from N = 4 to N = 5, a key keeps its server only if the old and the new remainder are equal. The pattern repeats every 20 keys, and the remainders match only for keys 0, 1, 2 and 3, which is 4 in 20. So 80% of the keys change server. For a cache, that means losing almost all the warm content in one moment.

Consistent hashing was invented so that a change in the set of servers moves only a small fraction of the keys, about 1/N.

              0
        S1 *       * k1
     *               *
   k2 *               * S2
     *               *
        S3 *       * k3
             half

 a key belongs to the first server
 found going clockwise from its position

The same hash function gives a position on the circle to each server and to each key. In the classic design, each key belongs to the first server found going clockwise from the key's position. The rule depends only on positions, so when a server joins or leaves, only the keys in one stretch of the ring change owner.

Example: 4 equal servers share the keys evenly, and a 5th equal server joins. The only keys that move are the ones the new server now owns: 1/5. Each old server goes from 1/4 to 1/5, giving up 1/20 of the total, and 4 × 1/20 = 1/5.

Virtual nodes

If every server has only ONE position on the ring, the stretches (arcs) can be very different in size, and when a server leaves, its clockwise neighbour inherits its whole share. That neighbour now carries two shares and, if it also fails, passes three to the next one: a cascade of failures.

The solution is virtual nodes: each server appears tens or hundreds of times on the ring, at many positions. Then the arcs become uniform, a bigger server can receive more positions (more share), and when a server leaves, its keys are spread over many others instead of one.

Health checks and failover

A balancer that keeps sending requests to a dead server is worse than no balancer. So it must find out who is alive. This is a health check. When it discovers a problem and moves the work to the others, that is failover: the automatic transfer of the work of a failed component to a healthy one. Failover needs two things: redundancy (another component can take over) and detection (something notices the failure).

The two complete each other. Passive is immediate but needs traffic. Active works with no traffic and is the only one that can bring a server back, because nobody sends real requests to a server that is out.

A back end marked unhealthy is not deleted. The balancer just stops choosing it for new requests and spreads the traffic over the others. When the health check passes again, it rejoins the rotation with no manual step.

At which layer do you check?

A check only detects problems at the layer where it works. Opening a TCP connection proves that some process is listening on the port, nothing more. If the application answers HTTP status 500 to everything, a TCP check says "healthy". To notice application errors, the probe must make an HTTP request and verify the status code (and the body when needed).

How long until a dead server is removed?

In the worst case, detection time = probe interval × number of failures required + probe timeout. If probes go every 2 s, three consecutive failures are required and the timeout is tiny, a server that dies right after a good probe is removed 2 × 3 = 6 seconds later. During those 6 seconds it still receives requests. Short intervals detect faster but raise false alarms: that is the trade-off.

Thresholds also give the decision hysteresis (a delay that avoids changing the decision too easily). If a server that alternates between working and failing could enter and leave the pool at every probe, each entry would cost client requests. Requiring several consecutive successes before returning trades a slower return for stability.

To retry or not to retry

When an attempt fails, may the balancer send the same request to another back end? It depends on what the first back end saw.

Liveness, readiness and shallow checks

• Liveness asks: is the process stuck and in need of a restart? Readiness asks: can it receive traffic now? An instance that is still loading caches and opening database connections is alive but not ready. The balancer should skip it, and nobody should restart it.
• A deep health check (one that tests shared dependencies, such as running a query on the shared database) makes all back ends fail together when the dependency fails. Removing all of them does not help, and it also takes down routes that would still work (like cached pages). The check used by the balancer should be shallow: can THIS instance serve? Watch dependencies by other means.

Removing a server without losing requests

Connection draining does it in two steps. First, the balancer stops choosing the instance for new requests. Then it waits for the requests in progress to finish, up to a deadline. Only then is the process stopped. Retries do not replace draining, because non-idempotent requests cannot be repeated safely.

High availability and single points of failure

A single point of failure (SPOF) is any part of the system that has no substitute: if it stops, everything stops, however redundant the other parts are. The useful question for each component is: what happens if only this one fails? High availability is the work of finding SPOFs and duplicating them.

Here is the surprise: five application servers behind ONE load balancer. The balancer removes the SPOF of the application tier, but if it stands alone it becomes a new SPOF. So balancers are deployed in pairs or groups, with a way to make traffic reach the healthy one, like a virtual IP address that changes owner, or DNS. Redundancy in one tier does not make up for the lack of it in another.

Active-passive and active-active

The arithmetic of availability

In parallel (redundant components, the set works while at least one is up): the set fails only if ALL fail together. With independent failures, the unavailability of the set is the product of the unavailabilities. Two balancers of 99% each: unavailability 0.01 × 0.01 = 0.0001, so availability 1 − 0.0001 = 99.99%.

In series (both parts are required): the availability of the set is the product of the availabilities. The pair at 99.99% in front of ONE database at 99.9%: 0.9999 × 0.999 = 0.9989001, about 99.89%. It never exceeds the weakest part. Adding even more balancers barely helps. The gain is in making the database redundant.

Split brain

Two balancers in an active-passive pair watch each other with a heartbeat (a regular "I am alive" message). Suppose the link between them breaks while both are healthy. Neither can tell the death of its peer from a failure of the link, since in both cases the heartbeat stops. Both may decide "I must be the active one" and fight over the same resource, such as the virtual IP address. This is split brain. To prevent it you need information from outside the link: a quorum with a third participant (a witness) that decides who may stay active, or fencing, which guarantees the other node was really shut down.

TLS termination

TLS makes a secret tunnel between two ends. The one who is the end of the tunnel holds the certificate (the ID card of the site) and the private key (the secret that proves the ID is theirs). To terminate TLS is to be that end: you decrypt the data. Where should the tunnel end when there is a balancer? There are three ways.

passthrough:  client ═════════════════════> back end

termination:  client ═══════> balancer ───────> back end
               (HTTPS)                (plain HTTP)

re-encrypt:   client ═══════> balancer ═══════> back end
               (HTTPS)                (new HTTPS)

The main benefit of terminating at the balancer is operational: TLS (certificates, keys, versions, ciphers) is configured in a single place. The back ends get simpler and spend their CPU on the application. The price is that the internal stretch is no longer encrypted, unless you re-encrypt.

Consequences you must know

• The scheme gets lost. After termination the application sees only the internal connection, which is HTTP. If users browse over https:// but the application builds redirects that start with http://, this is the cause. The proxy reports the original scheme in the header X-Forwarded-Proto, and the application should use it. Since any client can send that header, accept it only from trusted proxies.
• Encrypting is not authenticating. If NGINX opens proxy_pass https://backend; with no proxy_ssl_* directive, it does not verify the back end's certificate. Someone inside the network who poses as the back end receives traffic they can decrypt. With re-encryption you must turn verification on explicitly (proxy_ssl_verify on;) and give the list of accepted certificate authorities.

NGINX configuration

NGINX (here, the open source version 1.30) is told what to do through a configuration file. To balance, you do two things: declare a named group of servers, and point a proxy directive at that name.

upstream api {
    server app1:3000;
    server app2:3000;
    server app3:3000;
}

server {
    location /api/ {
        proxy_pass http://api/;
        proxy_set_header Host $host;
    }
}

• upstream api { ... } declares a group of servers and gives it a name. The group alone receives no traffic.
• proxy_pass inside a location turns it into a reverse proxy. When the name in the URL is an upstream group, the destination is chosen among the group's servers by the balancing method.
• The default method is weighted round robin, and each server has weight 1 unless you write weight. Other methods (least_conn, ip_hash, hash, random) apply only when you write the directive inside the block. For ip_hash on IPv4 the key is only the first three octets of the address, so 192.168.1.10 and 192.168.1.200 (same /24 network) go to the same server, and for IPv6 it is the whole address.

Server parameters

So a group with server a weight=3;, server b; and server c backup; sends traffic to a and b in the proportion 3 to 1, and c only receives requests when both a and b are unavailable.

Passive checks and retries

Open source NGINX has passive checking only (the active health_check belongs to the commercial version). It judges a server by the real requests. When max_fails is reached within fail_timeout, the server is skipped for fail_timeout, and after that a real client request is used to try it again. While the server stays dead, this cycle repeats. Meanwhile proxy_next_upstream (default error timeout) passes the failed request to another server. Non-idempotent requests (POST, LOCK, PATCH) already sent are excluded from this retry unless you add non_idempotent.

Two details that bite

• The trailing slash of proxy_pass. If proxy_pass has a URI part after the destination name (even a single /), NGINX replaces the part of the request that matched the location with that URI. With location /api/ { proxy_pass http://backend/; }, a request for /api/users reaches the back end as /users. Without the slash (http://backend) the path arrives as /api/users.
• The Host header. NGINX rewrites Host with $proxy_host, the destination written in proxy_pass. An application that uses Host to choose the site or build links would see an internal name. proxy_set_header Host $host; passes the name the client really asked for.

Caddy configuration

Caddy (version 2.11) does the same work with a shorter file, called the Caddyfile. There is no separate block to declare the group: the directive reverse_proxy is reverse proxy and load balancer at the same time. You list the addresses on the same line, and options go in a block between braces.

example.com {
    reverse_proxy app1:3000 app2:3000 app3:3000 {
        lb_policy round_robin
        health_uri /health
        health_interval 5s
        lb_try_duration 5s
    }
}

Automatic HTTPS

When the address of a site is a public domain name (like example.com above), Caddy turns HTTPS on with no extra configuration. It gets the certificate from a certificate authority using ACME, renews it automatically and redirects HTTP to HTTPS. As a load balancer, that is TLS termination without managing certificate files.

Defaults that are different from NGINX

lb_try_duration 5s tells Caddy to keep choosing upstreams and trying again for up to 5 seconds. That hides the loss of a back end from the client, at the price of some waiting. For active checks, health_uri makes the proxy itself send requests to that path on each upstream, whether or not there is client traffic. An upstream that fails leaves the selection and returns when the probe succeeds again.

Putting it all together

Now we can read the whole story. One server is not enough, so we scale horizontally (more instances) and put a load balancer in front. The balancer is a reverse proxy that decides which equal copy serves each request. At layer 4 it decides per connection, at layer 7 per request with richer rules. It chooses by an algorithm, it avoids dead servers thanks to health checks, it terminates TLS in one place, and it needs to be duplicated so it is not itself a single point of failure.

Common mistakes to avoid

• Believing X-Forwarded-For blindly: read it from the right, skipping trusted proxies.
• Adding instances when the database is the real limit, or when the connection pools will not fit.
• Using plain round robin with servers of different speeds.
• Using IP hash when many users share one address.
• Relying on sticky sessions to protect state: use shared storage or tokens.
• Expecting Caddy to retry and check health with no configuration, or NGINX open source to do active checks.
• Retrying a POST that may have run already.
• Forgetting that a lone load balancer is a SPOF, and that dependent parts in series cap the availability.
• Encrypting to the back end but not verifying its certificate.