Scalability: Vertical vs Horizontal

1The core idea#

Scalability is just a system's ability to take on more load without falling over: more users, more requests per second, more data. The question is never "will traffic grow," it is "when it does, what do I change?" There are two answers.

Vertical scaling (scaling up): give one machine more muscle.

• More CPU cores, more RAM, faster disks, same single server, just beefier.
• Think moving from a 4-core box with 16 GB of RAM to a 64-core monster with 512 GB.
• Your code and architecture do not change.

Horizontal scaling (scaling out): add machines and spread the work.

• Instead of one giant server, run ten modest ones behind a load balancer.
• Each handles a slice of the traffic.
• Growth becomes "add another box," not "buy a bigger one."

The mental model I give juniors. Your restaurant is slammed:

• Vertical scaling is hiring one superhuman chef who cooks four times faster. Simple to manage, but there is a ceiling on how fast any human gets, and if that person calls in sick, you are closed.
• Horizontal scaling is hiring four normal chefs and splitting the orders. They need coordination, but you can keep adding cooks, and losing one just slows you down a bit.

Simplicity plus a hard ceiling, versus coordination plus near-limitless growth. That trade is the whole topic.

2How it works#

Vertical scaling is mechanically boring, which is a compliment.

• You resize the instance. On AWS that is often changing m5.large to m5.4xlarge and rebooting.
• Code does not change. Architecture does not change.
• The server comes back with more resources and serves more concurrent requests. Done.

Horizontal scaling is where the interesting engineering lives. You cannot just clone a server and hope. Something has to sit in front and decide which copy handles each request. That is the load balancer.

• A request comes in, the balancer picks a healthy server (round robin, least connections, whatever policy you set), and forwards it.
• If one server dies, the balancer stops sending it traffic.
• Add a server, the balancer starts using it.

For this to work cleanly, your servers must be stateless.

• No request should depend on landing on the same machine as a previous one.
• If server 2 holds a user's login session in local memory and the next request routes to server 3, that user is suddenly logged out.
• So you push shared state out of the app servers into a place all of them reach: a session store like Redis, a shared cache, the database.

Get statelessness right and scaling out becomes almost mechanical.

• Traffic doubles, you double the servers.
• That is the dream, and it is achievable. It just takes discipline up front.

3A worked example with real numbers#

Abstract trade-offs convince no one. Here is a concrete capacity calculation, the kind I would sketch on the whiteboard. Say we are building the API tier for a product that just hit 10 million daily active users.

Getting to the peak number.

• Each user makes about 20 API calls a day, so 200 million requests per day.
• Spread over 86,400 seconds, the average is roughly 2,300 requests per second.
• Traffic is never flat. Peak is usually 3x to 4x average, so design for 9,000 RPS at peak with headroom.

The single-box plan.

• A beefy m5.4xlarge (16 vCPU, 64 GB) running a tuned Node or Go service handles maybe 2,000 RPS before p99 latency climbs past 200 ms.
• To hit 9,000 RPS on one box you need roughly 4.5x that capacity.
• The next steps up are m5.12xlarge and m5.24xlarge (96 vCPU, 384 GB), the top of that family.
• The 24xlarge gets close to the ceiling of the line and costs about $4.60 per hour on-demand, roughly $3,300 a month, for one server with zero redundancy.
• If it reboots, 10 million users see errors.

The horizontal plan.

• Take the same m5.4xlarge at about $0.77 per hour doing 2,000 RPS.
• To serve 9,000 RPS you need 5 of them, and you run 7 to carry one failure and leave headroom.
• That is 7 x $0.77, about $5.40 per hour, roughly $3,900 a month.
• Slightly more than the giant box, but losing one server now costs 1/7 of capacity (about 14%) and zero downtime.
• You can go to 14 boxes for Black Friday by lunchtime, then scale back down at night. The single-box plan has no such move left: it was already near the top of the instance line.

4Trade-offs#

This is the part interviewers care about most. Anyone can define the terms. Can you reason about what you give up?

A few of these deserve real talk.

The ceiling is real and closer than you think.

• The largest cloud instances today top out in the low thousands of GB of RAM and a few hundred vCPUs.
• In the worked example, I was already brushing the top of the m5 line just to serve 9,000 RPS on one box.
• That sounds like room until your product hits the front page and traffic goes 50x overnight.
• You cannot buy your way past physics. There is no server with a million cores.

Cost is not linear when you scale up.

• Doubling a small instance is cheap. At the top end, each step up costs more than the capability it adds.
• In the example, the 24xlarge was 6x the price of the 4xlarge for roughly 5x the vCPUs, and it handed me a single point of failure on top.
• The jump from a large box to a huge one often costs 3x for maybe 2x the throughput.
• A fleet of commodity boxes usually gives you more compute per dollar past a certain size, plus the resilience comes free.

Fault tolerance is the one I would lead with.

• A single scaled-up server is a single point of failure, and hardware does die.
• When it dies, your whole service is down until it reboots or you fail over.
• With 7 servers, losing one costs about 14% capacity and zero downtime.
• For anything users depend on, this alone often decides it.

5Where it is used#

Vertical scaling shows up where splitting the workload is genuinely hard.

• The classic case is a traditional relational database. Sharding a Postgres or MySQL primary across machines is painful: joins across shards, transactions that span shards, rebalancing data, all nightmares.
• So the common move is to scale the database up as far as you reasonably can, and only shard when truly forced to.
• You also see vertical scaling early in a product's life, when you have a handful of users and adding architectural complexity would be a waste of time.

Horizontal scaling is the default for stateless web and application servers.

• API backends, web frontends, worker queues: built to run as many identical copies.
• Big consumer services run thousands of app server instances, all interchangeable.
• It is also how distributed databases like Cassandra, DynamoDB, and modern object stores are built from the ground up: add nodes, get more capacity and more resilience.

My rule of thumb.

• Scale the stateless tier horizontally without a second thought.
• Scale the stateful tier vertically for as long as you can get away with it, before you are forced into the harder work of partitioning data.

6Common pitfalls and gotchas#

Storing session state on the app server.

• The number one reason horizontal scaling breaks: login sessions, uploaded files, in-memory caches living on individual boxes.
• The fix is to externalize state: sessions in Redis, files in object storage, cache in a shared layer.
• Design stateless from day one and your future self will thank you.

Forgetting the database is still shared.

• I have watched candidates add ten web servers, declare victory, and never notice every one of them hammers a single database.
• In the worked example, 7 app servers at 9,000 RPS still point at one Postgres primary. You just moved the bottleneck.
• Now the database is the wall. The data tier needs its own plan: read replicas, caching, eventually sharding.

Treating "scalable" as "infinite and free."

• More nodes means more network calls, more coordination, more things that fail independently, and a bigger bill.
• Distributed systems have real overhead. Do not scale out a system a single right-sized box would serve happily.

Ignoring the sticky-session crutch.

• Some teams patch the stateless problem by pinning each user to the same server (sticky sessions).
• It works until that server dies and takes those users' state with it, and it makes load uneven.
• It is a band-aid, not a design.

Over-engineering for scale you do not have.

• The flip side. 200 users do not need a multi-region auto-scaling fleet.
• Interviewers notice when you reach for complexity the problem does not justify. Scale when the numbers tell you to.

7How it shows up in interviews#

The scaling question almost always arrives the same way. You sketch a clean design, the interviewer nods, then says: "Great. Now it is 10 million users a day. What changes?"

What they are probing.

• Do you reach for horizontal scaling and a load balancer naturally?
• Do you understand why the app servers must be stateless, not just that they should be?
• Do you notice the database becomes the next bottleneck?
• Can you weigh cost, failure, and complexity with real numbers instead of reciting definitions?

What the answers sound like at each level.

The strong answer, spelled out. "At 10 million daily users with roughly 20 calls each, that is about 2,300 requests per second average, call it 9,000 at peak. I would scale the web tier horizontally behind a load balancer, say 7 mid-size instances so one can fail with no downtime, and keep those servers stateless so sessions live in Redis and any request can hit any server. That gives me both capacity and fault tolerance, since losing one box just costs about 14% of throughput. The next bottleneck becomes the database, because all 7 app servers still point at one primary. I would add read replicas for read-heavy traffic and put a cache in front of it. For the write path I would scale the primary vertically for now, and only shard if write volume outgrows the biggest instance, because sharding adds real operational pain. Early on, when traffic is small, I would keep it simple with one decent instance and not pay the complexity cost until the numbers demand it."

That answer hits every axis: a capacity estimate from first principles, horizontal for stateless, vertical where partitioning is hard, the shared-state trap, the database as the real wall, and the judgment to not over-build.

Next, try the System Design Question walkthrough on designing a URL shortener, where this exact vertical-versus-horizontal decision plays out across the web tier, the cache, and the data store.