HLD · 01 System Design: HLD

What Is High-Level Design?

What Is High-Level Design?

A booking service passes every one of its tests. Then the whole company starts using it, we run three copies behind a load balancer, and it books the same room twice. Every class is right; the bug is between the boxes. That's what high-level design is for: which boxes exist, how they talk, where the truth lives, and what happens when traffic grows or a box dies. We build it for real, in Docker on one laptop, and measure every step.

Chapters

Cheat sheet

Code

LLD01's booking service, byte for byte, behind a small Python web server; three copies behind nginx; one shared Postgres database; k6 for the load tests. Python 3.14.10, nginx 1.31.8, PostgreSQL 16.8 and k6 2.3.0, every image pinned in compose.yaml, and every number in the video from a saved run:

What we measured, on one laptop with a small test database:
Three copies, bookings in each copy's memory: 91 of 100 overlapping pairs double booked (the other 9 landed on the same copy by luck)
Three copies, one shared database with a no-overlap rule: 0 of 100
The estimated peak, 35 requests a second, on one copy: p99 3.46 ms, none failed
Flat out: one copy about 1,460 requests a second, three copies about 3,300 (2.3 times, not 3)
One copy killed mid-run: 1,572 of 3,949 requests failed with one copy, 0 of 4,001 with three (32 waited about a second)

Sources

Transcript

This is the booking service from the first episode of our low-level design series. Every class is tested, and all eleven tests pass. Now the whole company starts using it, so we run three copies of it behind a load balancer. Ana books room 4B from ten to eleven. Copy one says yes: booking number one.

Ben books the same room from ten thirty. Copy two says yes: booking number one. Room 4B is double booked again. Every class is right. Every test passes.

The bug is between the boxes, and catching bugs like this before they happen is high-level design.

This is Just Known. From unknown, to known. Episode one of our high-level design series: what is high-level design? In the low-level design series, we zoomed into one box and designed the classes inside it. High-level design zooms the other way: out, until you can see the whole system.

Which boxes exist, and how they talk to each other. Where the data lives. And what happens when traffic grows, or when one of the boxes dies. Low-level design makes each box right. High-level design makes the boxes right together.

Here's what went wrong. Each copy keeps its bookings in its own memory. Copy one knows about Ana's booking. Copy two has never heard of it. The lock from the low-level design still works, but it only protects one copy.

And the load balancer spreads requests over the copies, so Ana and Ben usually land on different ones. We tried it a hundred times, on a hundred different days. Ninety one were double booked. The other nine only worked by luck: both requests happened to reach the same copy. With a single copy, it was zero.

The fix is to stop keeping the truth inside the copies. The bookings move into one shared database, and the database enforces the rule itself. This rule tells Postgres that two bookings of the same room may never overlap, whichever copy sends them. Back to back is still fine, because each booking includes its start time, but not its end. Run the same test again.

Copy one takes Ana's booking, and copy two refuses Ben's, because the database said no. A hundred tries, and zero double bookings. And in one of our tests, thirty bookings for the same slot arrive at the same moment. Exactly one gets in. And notice what didn't change.

The new class offers the same two methods the web layer uses, book and free rooms, with the same signatures. The web layer calls them exactly as before. That's the low-level design paying off inside a high-level change. The box behind the interface moved, and nothing above it noticed. Zoom out a little further, and most systems you'll ever design are built from the same few boxes.

Clients, like a browser or a phone app. DNS, which turns a name into an address. A load balancer, which spreads requests over copies of the app servers. And a database, the source of truth. Then three more, for when the database isn't enough: a cache, for answers you give again and again.

A queue, for work that can happen later, like sending a confirmation email. And object storage with a CDN in front, for big files like photos and video. Each one gets its own episode. Our booking system uses the first five so far. Follow Ana's request through the system.

Her browser asks DNS where the booking site lives. The load balancer takes the request and picks a copy, say copy two. Copy two asks the database which rooms are free, and the answer travels back the same way. Every arrow on this diagram is a network hop that can be slow. Every box is a machine that can fail.

High-level design is about the arrows and the boxes. In an interview, high-level design follows a path. Requirements first: what the system must do, and how well. Then rough estimates of the load. Then the API and the data model, and then the diagram.

Then you go deep on the two or three parts that matter most, and you finish with the bottlenecks: what breaks first, and what you'd do about it. Let's walk our booking system along that path. The functional requirements say what the system does: book a room, see which rooms are free, cancel a booking. Those are the same as in the low-level design. The non-functional requirements say how well it must do them, and that's where high-level design lives.

It must never double book. It must stay up through the whole working day. Pages must answer fast. And it serves the whole company, say fifty thousand people. Each of those shapes the diagram more than any class does.

Now the estimate, on the back of an envelope. Fifty thousand people. Say each one opens the booking page six times a day. That's three hundred thousand requests a day. A day has eighty six thousand four hundred seconds, so on average, about three and a half requests a second.

But everyone books at nine in the morning, so let's say the peak is ten times the average: about thirty five requests a second. The amber numbers are guesses, and that's fine. An estimate tells you the size of the problem, not the exact answer. Steps three and four are short for a system this small. The API needs three calls: ask which rooms are free for a time and a number of people, book one, and cancel one.

Our test service has the first two. A booking that clashes gets the answer four oh nine, the HTTP code for a conflict. The data model is two tables, rooms and bookings, with our no-overlap rule on bookings. And step five, the diagram, is the one we've been drawing all along. Then we test the estimate for real.

A load tester sends thirty five requests a second to one copy, for a whole minute. Ninety nine percent of them were answered within about three and a half milliseconds, and none failed. That ninety nine percent mark is called the p99, and episode four is all about why it matters more than the average. Then we let it run flat out. One copy handled about fourteen hundred and sixty requests a second, roughly forty times our peak.

That's on one laptop with a tiny test database, so real requests would be heavier, but the gap is huge. Three copies reached about three thousand three hundred: a bit more than twice as many, not three times. Every box here, even the load tester, shares one laptop, so adding copies didn't add any machines. So for speed, one copy is plenty. The reason to run three is failure.

We send a steady hundred requests a second. About twelve seconds in, we kill the copy, and about fifteen seconds later we start it again. With one copy, the system is simply down: fifteen hundred and seventy two requests failed. Now the same test with three copies, killing copy one again. Not one request failed.

Thirty two of them waited about a second, while the load balancer worked out that copy one was gone. The other two copies carried the load. Here's what the load balancer did. When copy one stopped answering, nginx gave up on it after one second, and sent that same request to the next copy instead. Then it left copy one alone for five seconds at a time before trying it again, and each time it tried, a request or two waited that second.

That's where the thirty two slow requests came from. That was safe here, because these requests only read. A booking is different. If a copy dies halfway through one, nginx won't send it again by itself, because doing it twice could book the room twice. Making retries safe has a name, idempotency, and it's the subject of episode five.

Every box we add solves one problem, and moves the question somewhere else. The copies survive a crash, so now the weak spot is the database. If it dies, every copy fails at once. That's what replicas are for, in episode fourteen. And more boxes cost more: more machines, more to monitor, and more ways to fail.

For a tool used by a ten-person team, one server and a good backup is the right design. High-level design is choosing which problems you're willing to have. Designing for failure isn't paranoia. In a talk in 2009, Jeff Dean of Google listed what goes wrong in a new cluster's first year: about twenty racks failing, each taking dozens of machines with it, around a thousand individual machine failures, and thousands of failed hard drives. Leslie Lamport put it best, in an email back in 1987: in a distributed system, a computer you didn't even know existed can fail, and stop yours from working.

High-level design is how you build so that it doesn't. Interview corner. Design a meeting-room booking system for a company of fifty thousand people. Pause here, and sketch your answer in thirty seconds.

Here's a strong answer. Requirements: book, search, and cancel; never double book; and stay up all day. The estimate: about thirty five requests a second at the morning peak, which one server can handle. So the reason for several app servers is availability: identical copies behind a load balancer, with no bookings kept inside them. The truth lives in one database that refuses overlapping bookings itself, so no copy can get it wrong.

Then name the next steps: a replica for the database, a cache for the free-rooms page, and a queue for the confirmation emails. To stand out, say why each box is there. Three copies because one can die, not three copies because of scale. Let's recap. High-level design is the whole system at once: the boxes, the arrows between them, and where the truth lives.

You estimate before you build, and then you measure. And you design for failure, not just for load. Here's the road ahead. Act one: thinking at scale, ending with a URL shortener. Act two: the building blocks, like load balancers, caches, and queues.

Act three: data at scale. Act four: reliability, ending with a payment system. Act five: system design for mobile apps. Act six: the classic interview case studies. And to finish, a full mock interview.

Next time, we get much better at the back of the envelope: how to estimate traffic, storage and bandwidth before you build anything. That's episode two. This is Just Known. From unknown, to known.

Note: Ana, Ben, the rooms and the company are made-up examples; the 50,000 people, 6 views a day and the 10x peak are assumptions for the estimate. Every request, test and measurement on screen is from a real run on one laptop (Intel Core i7-13700HX, Docker Desktop), so the timings show the shape of the results, not production numbers. Pictures marked "illustration" are not real output.
Narration voice generated with Google Gemini text-to-speech.