How Load Balancers Split Traffic Between Thousands of Servers
By Devang Pandya · Cloud Tech by Devang · Cloud • DevOps • AI — Made Simple
When you open a big website, your request almost never goes straight to "the server". It goes to a load balancer first. The load balancer decides which of many servers should handle your request. It does this for every request, millions of times a second, and you never notice it. That's the point.
Load balancers are one of the most important building blocks in cloud architecture. They give you scale (many servers), availability (broken servers are skipped) and flexibility (you can add, remove or update servers without users noticing).
In this post I will explain, in simple English, what a load balancer does, the main ways it picks a server, how health checks work, and the difference between Layer 4 and Layer 7 load balancers, including AWS's NLB and ALB. At the end you will get a clean answer for the classic interview question: "L4 vs L7 load balancer: when do you use which?" I have spent 14+ years in cloud, DevOps and SRE, and I have seen many outages that came down to one misconfigured health check, so we will spend time on that too.
1. What a load balancer does
A load balancer sits between clients (browsers, mobile apps, other services) and a group of servers. In AWS language the servers are targets, grouped into a target group. In NGINX language they are an upstream.
Its main jobs:
- Single front door. Clients talk to one name or IP address. They don't need to know how many servers there are.
- Spread the load. It sends each request (or connection) to one of the healthy servers.
- Skip broken servers. It checks server health and stops sending traffic to servers that fail.
- Enable scaling. You can add servers when traffic grows and remove them when it drops. AWS Auto Scaling groups can register new instances with a target group automatically.
- Often more: TLS termination, routing rules, sticky sessions, and protection from some attacks.
The AWS documentation sums it up well: the load balancer is a single point of contact for clients, and it distributes incoming traffic across its healthy registered targets.
2. How does it pick a server?
This is called the load balancing algorithm. The most common ones are below.
Round robin
Send requests to servers in turn: 1, 2, 3, 4, then back to 1. It is simple and fair when all servers are similar and requests cost about the same. Round robin is the default for AWS Application Load Balancer target groups and for NGINX.
Least connections
Send the next request to the server with the fewest active connections, the least busy one. It works better when some requests take much longer than others (for example file uploads or reports). NGINX supports this as least_conn.
On AWS ALB, the similar option is called least outstanding requests. It sends the request to the target with the fewest requests still in progress.
Other methods you may meet
- Weighted: give stronger servers a bigger share (for example 70/30 during a canary release). ALB also offers weighted random.
- IP hash / consistent hashing: the same client (or the same key) always goes to the same server. This is useful for caches.
- Least time: pick the server with the fastest recent responses (NGINX Plus).
Lesson for us: start with round robin. Switch to least connections or least outstanding requests when request times vary a lot. Use hashing only when you really need the same client to keep hitting the same server.
3. Health checks: never send users to a broken server
A load balancer is only as good as its health checks.
A health check is a small, regular test. For example: "Every 10 seconds, call GET /health on each server. If it fails a few times in a row, mark it unhealthy. If it passes a few times in a row, mark it healthy again."
The load balancer only routes to healthy targets. When a server crashes, hangs or is being replaced, traffic moves to the others automatically.
Some useful related features on AWS:
- Deregistration delay (connection draining): when you remove a target, the load balancer stops sending it new requests but lets in-flight requests finish. The ALB default is 300 seconds.
- Slow start: a new target gets a slowly increasing share of traffic instead of a full share at once, so a cold app or cache isn't flooded.
- Target group health thresholds: you can say "if fewer than N targets are healthy, fail over" instead of relying on just one healthy server.
Common mistakes I see in real systems:
- Health check too shallow. It returns 200 even when the app cannot reach its database, so users get errors from "healthy" servers.
- Health check too deep. It fails when an optional dependency is slow, so all servers go unhealthy at once and the whole site goes down.
- Thresholds too slow or too twitchy. Too slow and users hit dead servers. Too twitchy and servers flap in and out.
A good health check answers one question: "Can this server do useful work right now?"
4. Layer 4 vs Layer 7
"Layer" refers to the OSI model of networking. You only need two layers here.
Layer 4 (transport layer)
A Layer 4 load balancer works with TCP and UDP. It sees IP addresses and ports, but it does not read the content of your HTTP request. It forwards connections.
- Very fast and handles huge volumes.
- Works for any TCP/UDP protocol: databases, MQTT, gaming, custom protocols, not just HTTP.
- Cannot route by URL path or headers, because it doesn't look at them.
On AWS this is the Network Load Balancer (NLB). AWS describes it as working at the fourth layer of the OSI model and able to handle millions of requests per second.
Layer 7 (application layer)
A Layer 7 load balancer understands HTTP/HTTPS (and often gRPC). It reads the request: path, host name, headers, query strings, methods.
- Smart routing:
/api/*to the API service,/images/*to the image service,admin.example.comto the admin app. - Can do TLS termination, redirects, fixed responses, authentication integration, and sticky sessions with cookies.
- A bit more processing per request than Layer 4.
On AWS this is the Application Load Balancer (ALB). AWS describes it as working at the application layer, the seventh layer of the OSI model. It evaluates listener rules in priority order to decide which target group gets each request.
Side by side
| Layer 4 (e.g. AWS NLB) | Layer 7 (e.g. AWS ALB) | |
|---|---|---|
| Sees | IP, port, TCP/UDP | Full HTTP request: path, host, headers |
| Routing | By connection | By content (path, host, header, method) |
| Protocols | Any TCP/UDP | HTTP, HTTPS, gRPC |
| Strength | Raw speed, any protocol | Smart routing, HTTP features |
| Typical use | Non-HTTP services, very high throughput | Web apps, APIs, microservices |
In many real systems you use both: for example an NLB at the edge for raw performance and a fixed entry point, and an ALB or an ingress controller (like NGINX) behind it for HTTP routing.
5. Sticky sessions: useful, but be careful
Sticky sessions (session affinity) send the same user to the same server, usually using a cookie. ALB supports this (it is off by default).
It can help with old apps that keep sessions in server memory. But it fights against good load spreading, and if that server dies the user's session is lost. The better long-term fix is to make app servers stateless and store sessions in Redis or a database. (I cover this in topic 004 of this series, on how Instagram scaled.)
6. Where load balancers fit in a bigger design
A typical web architecture on AWS:
Users
│
▼
DNS (Route 53)
│
▼
CloudFront (CDN, optional)
│
▼
Application Load Balancer ── health checks ──► unhealthy targets skipped
│ │ │
▼ ▼ ▼
App App App (Auto Scaling group, stateless)
│
▼
Database / cache
And inside Kubernetes, the same ideas show up as Services (Layer 4 spreading across pods) and Ingress or Gateway API (Layer 7 routing).
7. The example visuals
The "srv-1 to srv-4" boxes, the broken server and the "1, 2, 3, 4, repeat" labels in the Reel are examples to show the idea. Real fleets can have a few servers or thousands. The algorithms and NLB/ALB layers are accurate. Note that the Reel uses the general term "least connections". On AWS ALB the closest option is called "least outstanding requests".
8. How to answer this in an interview
Question: "L4 vs L7 load balancer: when do you use which?"
Strong answer:
"A Layer 4 load balancer works at the TCP/UDP level. It routes connections by IP and port without reading the request, so it's very fast and works for any protocol. On AWS that's the Network Load Balancer. A Layer 7 load balancer understands HTTP, so it can route by path, host or header, terminate TLS and do things like redirects and sticky sessions. On AWS that's the Application Load Balancer. I use Layer 4 for raw speed or non-HTTP traffic, and Layer 7 for web apps and microservices that need smart routing. Either way, health checks are critical: the load balancer must only send traffic to targets that can really do work."
Follow-ups to expect:
- Round robin vs least connections? Round robin for similar, short requests. Least connections (or least outstanding requests on ALB) when request times vary.
- How do you deploy without dropping requests? Use connection draining (deregistration delay), roll servers one batch at a time, and use slow start for new ones.
- What happens if all targets are unhealthy? Depending on settings, the load balancer may fail open (send to all) or fail over via DNS. Know your configuration before an incident.
- Is the load balancer a single point of failure? Managed cloud load balancers run across multiple Availability Zones. For self-managed ones you run at least two, with failover.
Sources
- AWS, "What is a Network Load Balancer?" (Layer 4, millions of requests per second): https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html
- AWS, "What is an Application Load Balancer?" (Layer 7, listener rules): https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html
- AWS, "Target groups for your Application Load Balancers" (healthy targets only, round_robin default, least_outstanding_requests, weighted_random, deregistration delay default 300s, slow start, stickiness, target group health): https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html
- NGINX, "HTTP Load Balancing" (Round Robin, Least Connections, Least Time, IP Hash, Generic Hash): https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/
Preparing for a Cloud, DevOps or SRE interview? Book a Free 15-min Cloud & DevOps Interview Guidance Call with me: what to study, how to answer system-design and on-call questions, plus a CV and roadmap check 👉 topmate.io/cloudtechbydevang
About the author
- I help engineers learn cloud, DevOps, SRE and AI in simple English, crack interviews and follow clear roadmaps. 14+ years building and running platforms on AWS, GCP, Kubernetes and Terraform. One topic a day.
Comments
Join the discussionNo comments yet. Be the first to comment.
