SDC#21 - How to Scale a Component?
Interview Tips on Vertical vs Horizontal Scaling and more...
Hello, this is Saurabh…👋
Welcome to the 235 new subscribers who have joined us since last week.
If you aren’t subscribed yet, join 3000+ curious Software Engineers looking to expand their system design knowledge by subscribing to this newsletter.
In this last edition of 2023, I cover the following topics:
🖥 System Design Concept → How to Scale a Component?
🍔 Food For Thought → Moving Out of Cloud
So, let’s dive in.
“How do you scale an individual component in your system?”
This is a popular interview question that’s meant to spark a discussion of some sort.
How do you answer this question?
While there are several possible answers depending on the exact context of the problem, there are some bare minimum points you should address.
Let’s go through them one-by-one.
Why do you need to Scale?
Some common reasons are:
Handling a greater workload
Serving more users
Processing more requests
As the user base of your application grows, typically it needs to handle a greater workload.
Greater workload usually means dealing with more users and processing a higher number of requests.
The need to scale a component might apply to a backend service. It could also be a web application. Or even a database or cache.
Most of the time, the requirement to scale is a good sign that your application or product is doing well. Sometimes, however, it can also be an unintentional bug that’s causing your application to guzzle up resources, forcing you to think about scaling to meet the existing demand.
How do you scale a component?
The easiest approach is Vertical Scaling.
It’s quite simple to understand as it generally involves thinking bigger.
More RAM. A bigger disk. Or a bigger CPU.
With Vertical Scaling, you are basically trying to upgrade the hardware running the component. This isn’t so different from how you upgrade your PC to play the latest video game on the market.
Vertical Scaling can quickly address your scalability needs.
Most of the cloud providers provide easy options to increase the capacity (CPU, memory or disk) of your instance.
A major advantage of Vertical Scaling is that you don’t run into any issues related to distributed data.
But there are two main disadvantages as well:
Vertical Scaling is not cost-effective in the long run
A single component instance, no matter how powerful it is, can still become a single-point-of-failure (SPOF)
This brings us to the second approach for scaling a component.
It’s called Horizontal Scaling. Also, scaling out.
With horizontal scaling, you increase the capacity of a component by adding more machines.
For example, consider that your database hosted on a single 8 vCPU instance with 32 GB RAM runs out of capacity.
You scale out the database by partitioning it over two additional server nodes, each with 8vCPU and 32 GB RAM. Now, each machine has the same individual capacity but your workload is spread across three nodes.
This is a lot like the Divide-and-Conquer approach that’s used to solve many computing problems.
Here’s an illustration that shows the difference between horizontal vs vertical scalability:
Now, horizontal scaling has a couple of flavors depending on whether a component is Stateless or Stateful.
Let’s look at both:
1 - Stateless Component
A stateless component doesn’t save the client data from one session for use in the next session.
This makes them easier to scale horizontally.
You can simply add more instances to your infrastructure to handle an increase in load. Similarly, you can easily remove the unused instances to optimize resource utilization and save costs.
See the below diagram that shows this arrangement:
But how do you make sure that all instances are utilized efficiently?
As shown in the diagram, you can install a load balancer in front of the application instances.
The load balancer ensures that incoming requests are distributed optimally between the various instances of the stateless component.
If you are interested about load balancers, I wrote a detailed post on it last week.
2 - Stateful Component
A stateful component or application saves data from one session for use in the next session.
This could be a web application storing session information. It could also be a database or a key-value store.
Horizontally scaling a stateful component is somewhat trickier.
For example, to horizontally scale out a database instance , you need to replicate the data to multiple nodes. One of these nodes is the primary and the other are replicas.
The primary node takes care of the write requests and the replica nodes take care of the reads.
See the below diagram for reference:
Do note that this isn’t the only possible replication arrangement. You could have multiple combinations that I’ll cover in a future post.
Interview Points for Scalability
Having understood the high-level approaches of scaling a component, the question remains.
How should you tackle questions around scalability in an interview setup?
Here are a few points that can help you give a balanced answer:
The general viewpoint is that Vertical Scaling is less complex. You should stay with Vertical Scaling for as long as possible because it can be sufficient for a vast majority of applications. Of course, you’ve to deal with SPOFs by maintaining some backups.
When it becomes obvious that Vertical Scaling is no longer cost-effective, move to Horizontal Scaling. You’d know the threshold has been crossed when adding more hardware resources doesn’t provide a proportional performance benefit.
Either way, you should try to build your applications in a stateless manner. When moving from vertical to horizontal scaling, it will be easier to deal with such applications.
In case of stateful components such as databases, go with the databases that support horizontal scalability. Most modern databases do so but it’s easier to ignore the setup complexities.
If you have any other points that come to mind, do mention in the comments section below.
🍔 Food For Thought
👉 Moving out of Cloud
A recent post by DHH has caused quite a storm when he announced they have moved out of Cloud completely and ended up saving a ton of money in the process.
What do you think about this move? And is it the beginning of a trend that’s going to accelerate?
Here’s the link to the post:
https://x.com/dhh/status/1736832920370000085?s=20
👉 Specialization vs Flexibility
Should you master one skillset or should you have diverse knowledge of many areas?
This is a constant debate that every software engineer goes through.
Often, a mixed approach works far better than choosing one over the other.
The below tweet summarizes this viewpoint quite well.
Here’s the link to the tweet:
https://x.com/RaulJuncoV/status/1736032652509995160?s=20
That’s it for today! ☀️
Happy holidays and happy new year to all the wonderful subscribers.
Enjoyed this issue of the newsletter?
Share with your friends and colleagues
See you in 2024 with another value-packed edition — Saurabh
nice article. for completeness I just want point out a couple of things:
- the database replication is a pattern called "read replicas" and together with sharding (splitting the db by rows) and federation (splitting the db by table) are some great techniques to scale a database
- on the read path to the db you can also add an in memory cache like Redis and a CDN for static assets.
if you are interested I wrote an article about the role of CDNs in system design at https://cloudnativeengineer.substack.com/p/the-role-of-content-delivery-networks