But with increased complexity comes greater risk.

If you are building distributed systems, you must invest in proactive strategies to ensure resiliency. These strategies can be divided into two categories:

• Downstream Patterns

• Upstream Patterns

Let’s look at both.

Downstream Resiliency Patterns

Downstream patterns are used by the service caller. They are implemented by a service when it communicates with another service, ensuring that a failure in the downstream system doesn’t cause a domino effect.

1 - Timeouts

A timeout is a mechanism that prevents a service from waiting indefinitely for a response from a downstream service. By setting a maximum wait time, you avoid tying up resources that could be used for other requests.

Why It’s Important:

• Prevents resource exhaustion caused by requests stuck waiting indefinitely.

• Improves overall system responsiveness by allowing failed requests to be handled quickly.

Implementation Tips:

• Fine-tune timeouts based on the expected response times of downstream services. For example, a cache lookup might have a 50ms timeout, while a database query might have a 500ms timeout.

• Combine timeouts with monitoring to identify latency patterns and adjust as needed.

2 - Circuit Breaker

A circuit breaker monitors the success and failure rates of requests and temporarily disables calls to a service if failures exceed a predefined threshold. Like an electrical circuit breaker, it prevents further damage by “tripping” the connection.

Why It’s Important:

• Protects your system from overwhelming downstream services during failures.

• Allows time for the downstream service to recover.

Implementation Tips:

• Implement with standard libraries like Resilience4j to handle circuit breakers.

• Define three states for the circuit:

  1. Closed: Requests flow as usual.

  2. Open: Requests are blocked after reaching a failure threshold.

  3. Half-Open: A limited number of test requests are sent to check if the downstream service has recovered.

• Tune thresholds carefully, such as the number of failures or the time window for evaluating failures.

3 - Retries with Exponential Backoff

Retries are essential when transient failures occur.

Exponential backoff adds a delay between retries, doubling the wait time after each attempt. This gives the downstream service more time to recover and avoids overwhelming it.

Why It’s Important:

• Increases the likelihood of success when the failure is temporary.

• Reduces the risk of overloading a struggling downstream service.

Implementation Tips:

• Combine retries with a maximum retry limit to avoid infinite loops.

• Use jitter (randomized delay) in conjunction with exponential backoff to prevent synchronized retries from multiple clients causing a “thundering herd” problem.

• Example: First retry after 100ms, second after 200ms, third after 400ms, with a maximum of 5 retries.

Upstream Resiliency Patterns

Upstream patterns are used by the service owner. They are implemented by service owners to safeguard the health of their service and ensure stability during high traffic or failures.

1 - Load Shedding

Load shedding involves rejecting a portion of incoming requests when a service is overloaded.

Why It’s Important:

• Prevents complete system failure during peak loads.

• Ensures critical users or requests are prioritized.

Implementation Tips:

• Use techniques like token-bucket or leaky-bucket algorithms to control the rate of incoming requests.

• Implement priority queues to ensure critical requests (e.g., payment transactions) are handled first.

• Gradually degrade non-critical features or services under heavy load (e.g., disabling analytics during peak demand).

2 - Rate Limiting

Rate limiting sets the maximum number of requests that a client can make within a specific time window. This prevents individual clients or malicious actors from overwhelming your service.

Why It’s Important:

• Protects the service from abuse, such as Distributed Denial of Service (DDoS) attacks.

• Ensures fair resource usage across all clients.

Implementation Tips:

• Implement rate-limiting algorithms like the fixed window, sliding window, or token bucket.

• Use APIs or middleware solutions such as API Gateway, NGINX, or tools like Kong.

• Provide informative error messages (e.g., HTTP 429 Too Many Requests) to guide clients on retry strategies.

3 - Bulkheads

Bulkheading involves isolating different parts of the system to prevent failures in one component from cascading to others. This is analogous to compartments in a ship—if one is breached, the others remain unaffected.

Why It’s Important:

• Limits the blast radius of failures.

• Improves fault isolation and prevents a single failure from affecting the entire system.

Implementation Tips:

• Separate resource pools for different components (e.g., thread pools or connection pools for different services).

• Isolate critical resources to prevent non-critical services from consuming them (e.g., separate database connections for high-priority and low-priority queries).

• Use container orchestration tools like Kubernetes to allocate dedicated resources to specific services.

4 - Health Checks with Load Balancers

Health checks monitor the status of service instances and enable a load balancer to redirect traffic away from unhealthy instances. This ensures that users are always routed to functioning parts of the system.

Why It’s Important:

• Prevents traffic from being sent to failing or degraded instances.

• Improves overall user experience by minimizing downtime.

Implementation Tips:

• Use active health checks (periodic pings) or passive health checks (monitoring error rates).

• Configure thresholds for health checks, such as the number of failed requests or response latency.

• Implement rolling updates with health checks to ensure smooth deployment without downtime.

So - which resiliency patterns have you used in your project?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

In software development, the term BFF, which stands for Backends-for-Frontends, refers to a design pattern where dedicated API gateways are created for each device or interface type.

For example, you create a separate BFF for web browsers, mobile apps, or public/partner APIs.

Each BFF acts as a specialized layer between the client and the underlying services, tailoring the API to the specific needs of the client.

Key Characteristics of BFFs

1. Dedicated API Gateway: Each BFF serves as a dedicated API gateway for a specific device or interface type. It encapsulates the logic and requirements specific to that client type.

2. Customized Functionality: BFFs can handle various functionalities beyond simple API routing. They can perform tasks such as:

   1. rate limiting,

   2. authentication,

   3. header sanitization

   4. cache control

3. Decoupling from Lower-Level Services: By implementing BFFs, the logic specific to different client types is abstracted away from the lower-level services. This allows the services to focus on their core functionality while the BFFs handle client-specific requirements.

Advantages of the BFF Pattern

Implementing the BFF pattern offers several advantages:

1. Resiliency: By having separate BFFs for different client types, the impact of a faulty deployment or issue in one BFF is isolated to that specific client. Other clients remain unaffected, improving overall system resiliency.

2. Autonomy: Each BFF can be optimized according to the specific needs of the client it serves. For example, mobile clients may prefer larger responses to minimize network calls, while web clients may prioritize smaller payloads. BFFs provide the flexibility to cater to these varying requirements.

3. Development Speed: The resiliency and autonomy provided by BFFs boost developer confidence and enable faster development velocity. Teams can work independently on different BFFs without the fear of impacting other clients.

Disadvantages of the BFF Pattern

While the BFF pattern offers benefits, it also has some drawbacks to consider:

1. Code Duplication: When multiple BFFs share common functionalities, such as authorization or business logic, there is a risk of code duplication across the BFFs. This can lead to maintenance challenges and inconsistencies.

2. Complexity: As the BFFs become responsible for more glue-logic and orchestration of services, they can become more complex over time. Managing this complexity requires careful design and maintenance.

3. Proliferation of BFFs: Developers may be tempted to create a new BFF for every slightly different scenario, leading to an excessive number of BFFs. This can result in increased complexity and maintenance overhead.

Best Practices for Implementing BFFs

To effectively leverage the BFF pattern, consider the following best practices:

1. Clearly Define Client Requirements: Thoroughly understand the specific requirements of each client type and design the BFFs accordingly. Avoid creating unnecessary BFFs for minor variations.

2. Minimize Code Duplication: Identify common functionalities across BFFs and consider extracting them into shared libraries or services to minimize code duplication.

3. Maintain Clear Boundaries: Clearly define the responsibilities of each BFF and ensure they align with the overall system architecture. Avoid overloading BFFs with excessive functionality.

4. Monitor and Optimize Performance: Regularly monitor the performance of BFFs and optimize them based on client requirements and usage patterns. Ensure efficient resource utilization and scalability.

So - have you used the BFF pattern?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

I’m talking about the TCP 3-way handshake.

Every time you open a website, TCP makes sure this handshake happens and you get a reliable connection.

How does TCP pull it off?

Let’s look at each step:

1 - SYN (Synchronize)

• Think of it as one party extending their hand for a handshake.

• The process starts with the client sending a TCP segment to the server with the SYN flag set.

• Also, the client selects an initial sequence number (ISN) and includes it in the SYN segment. Let’s say the ISN is 1000.

• This tells the server that the client wants to establish a connection.

2 - SYN-ACK (Synchronize Acknowledge)

• This is like the second party taking the hand.

• The server receives the SYN segment and responds by sending a SYN-ACK segment to the client.

• Before sending, the server sets the SYN flag and the ACK flag in the segment

• The ACK flag is meant to acknowledge the receipt of the client’s SYN segment by incrementing the ISN by 1 (i.e. 1001)

• The server also selects its own ISN (let’s say 5000) and includes it in the SYN-ACK segment.

3 - ACK (Acknowledge)

• This is where the two parties shake hands.

• The client receives the SYN-ACK segment.

• The client acknowledges the server’s response by sending another TCP segment with the ACK flag set.

• The sequence number in the segment is set to the server’s ISN incremented by 1 (basically, 5001)

After this, the connection is established.

The diagram below shows all the three steps:

FAQs

A few additional FAQs about the entire TCP handshake business:

What’s the deal with SYN and ACK? What are these exactly?

In TCP, several control flags are used to manage different aspects of the communication process. These flags are bits in the TCP header.

Some common ones are SYN, ACK, FIN, RST, PSH, and so on.

Why sequence numbers (ISNs) are so important?

The numbers help the client and server keep track of the message order so that nothing gets missed.

Also, these numbers are randomized and help prevent malicious actors from hijacking the TCP session.

Is the Internet moving away from TCP?

HTTP/3 doesn’t use TCP but relies on QUIC (Quick UDP Internet Connections). So yes, we may see a gradual move away from TCP.

However, the majority of websites are still on HTTP/1.1 and HTTP/2. So nothing is going to happen tomorrow.

So - what do you think about the TCP 3-way handshake?

Is it an overkill? Or a great mechanism for establishing connections?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

At its core, OOP is based on four fundamental principles: Abstraction, Encapsulation, Inheritance, and Polymorphism.

Let’s break down each concept with explanations, examples, and best practices.

1 - Abstraction

Abstraction is the process of hiding complex implementation details while exposing only the essential features of an object. It allows developers to work with high-level concepts without worrying about the underlying implementation.

How It Works

Imagine a Car class with a method called stop(). Users of this class don’t need to know how the car stops—they just need to call stop(), and the underlying implementation takes care of the details.

abstract class Car {
    abstract void stop();  // Abstract method, no implementation
}

class Honda extends Car {
    @Override
    void stop() {
        System.out.println("Honda slowing down and stopping.");
    }
}

class Tesla extends Car {
    @Override
    void stop() {
        System.out.println("Tesla applies brakes and stops.");
    }
}

Here, Car defines the stop() method but doesn’t provide an implementation. Honda and Tesla provide their specific implementations, hiding the details from the users of these classes.

Why Abstraction Matters

• Simplifies complex systems by exposing only necessary parts.

• Improves code maintainability by separating the interface from implementation.

• Enhances security by hiding implementation logic from unauthorized access.

2 - Encapsulation

Encapsulation is the concept of bundling data (fields) and methods that operate on the data into a single unit (class) while restricting direct access to the data.

How It Works

Encapsulation is achieved using access modifiers (private, protected, public) to control access to variables and methods.

class BankAccount {
    private double balance;  // Private variable, cannot be accessed directly

    public BankAccount(double initialBalance) {
        this.balance = initialBalance;
    }

    public void deposit(double amount) {
        if (amount > 0) {
            balance += amount;
        }
    }

    public double getBalance() {
        return balance;  // Provides controlled access to balance
    }
}

Here balance is private, so it cannot be accessed directly from outside. Also, getBalance() is public, allowing controlled access to the balance.

Why Encapsulation Matters

• Protects data integrity by preventing unauthorized modifications.

• Reduces dependencies by keeping data and its operations in one place.

• Enhances flexibility as changes to implementation don’t affect external code.

3 - Inheritance

Inheritance allows a class (child) to inherit attributes and methods from another class (parent). This promotes code reuse and hierarchical relationships between classes.

How It Works

Consider a Vehicle class. Instead of rewriting common functionality for each type of vehicle, a Car and Bike can inherit from Vehicle.

class Car {
    String brand = "Generic Brand";

    void honk() {
        System.out.println("Beep! Beep!");
    }
}

class Car extends Vehicle {
    int doors = 4;
}

public class Main {
    public static void main(String[] args) {
        Car myCar = new Car();
        System.out.println(myCar.brand);  // Inherited from Vehicle
        myCar.honk();  // Inherited method
        System.out.println("Number of doors: " + myCar.doors);
    }
}

Here, Car inherits brand and honk() from Vehicle, reusing existing logic.

Why Inheritance Matters

• Reduces code duplication, making maintenance easier.

• Creates hierarchical relationships (e.g., Animal → Mammal → Dog).

• Improves extensibility, allowing new behaviors to be added to child classes.

4 - Polymorphism

Polymorphism allows a single method to behave differently based on the object it is called on. This means one function can be applied to multiple object types.

There are two types of polymorphism:

1. Compile-time

2. Run-time

How It Works

Here’s an example of polymorphism:

class MathOperations {
    int add(int a, int b) {
        return a + b;
    }

    double add(double a, double b) {
        return a + b;
    }
}

public class Main {
    public static void main(String[] args) {
        MathOperations math = new MathOperations();
        System.out.println(math.add(5, 10));  // Calls int version
        System.out.println(math.add(5.5, 2.2));  // Calls double version
    }
}

Here, the add() method behaves differently based on the parameter type.

Why Polymorphism Matters

• Enhances flexibility by allowing objects of different types to be treated uniformly.

• Encourages code extensibility without modifying existing code.

• Supports the Open-Closed Principle (OCP)—extend behavior without modifying existing logic.

So - have you used object-oriented programming? And are there any other concepts you find important?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

It hosts billions of posts. And a lot many of these posts contain media content such as images, videos, gifs, and so on.

While the media content is often stored in object storage, the metadata needs to be stored elsewhere. For example, if you’ve got a video, you might need to store information such as the thumbnail URL, playback URLs, bitrates, and various resolutions.

Reddit had a big problem with this metadata.

While their systems had the information, it was scattered across multiple systems. Media data used for traditional image and video posts was stored with post data, whereas media data related to chats and other types of posts was stored in a different database.

This made it difficult for them to analyze and categorize the metadata. Auditing changes became tough. Even checking information about a specific image or video required them to query the S3 bucket.

To solve these challenges, the Reddit Engineering team decided to build a media metadata store.

Requirements of the Metadata Store

There were several requirements for the new metadata store:

• Move all existing media metadata from different systems into a unified storage. In other words, migrate existing data to the new database.

• Support data retrieval to the tune of over 100K read requests per second with very low latency (less than 50 ms).

• Support media creation and updates.

After evaluating multiple choices, the Reddit Engineering Team went for AWS Aurora Postgres.

The diagram below shows a high-level architecture of their media metadata storage system.

The API service interfaces with the database to handle both reads and writes. To manage the connection pooling, they use pgBouncer as the proxy for connecting to Postgres.

Just so you know, pgBouncer is a lightweight connection pooler for Postgres.

Data Migration Process

While setting up a new Postgres database is easy, the real challenge lies in migrating terabytes of data from one database to another while serving 100K requests per second.

It’s like changing a car’s tires while you are driving it.

The consequences of the new database having the wrong media metadata could spell disaster for the website.

The migration was handled as follows:

• Enable dual writes into both the new and old databases from the clients of media metadata. This ensures that any new metadata is stored in the new database.

• Backfill data from older databases to the metadata store.

• Next, enable dual reads on media metadata from our service clients.

• Compare the output of the reads and fix any data gaps.

• Ramp up read traffic to the new database.

Here’s what the dual writes and backfill part looks like.

Dealing with Issues

While the overall approach is fine from a happy path perspective, that’s not always the case in real software development.

Some issues that could come up are as follows:

• Data transformation bugs in the service layer.

• Writes to the new metadata store could fail and the writes to the source database succeed resulting in inconsistent data.

• The backfill process during migration can overwrite newer data from the service write.

To handle these scenarios, the Reddit team built a more robust solution as shown below:

Here is what’s going on:

• The client dual writes to the new database and the old database.

• A Kafka consumer listens to the stream of data change events from the source database. This is done using a Change Data Capture (CDC) process.

• The consumer performs data validation with the media metadata store to check inconsistencies.

• In case of any inconsistency, the consumer reports the differences to another database so that engineers can take a look and fix the issues.

What about Scaling?

The new media metadata store was heavily optimized for reads, helping Reddit achieve a p50 latency of 2.6 ms and p99 latency of 17 ms at 100K requests per second.

However, the data volume is also quite high and according to their estimates, the media metadata can reach 50 TB by 2030.

To handle this scalability challenge, Reddit has implemented partitioning in Postgres using an extension known as pg_partman.

The above SQL statement is scheduled using the pg_cron scheduler to create new partitions when the number of spare partitions goes below a certain number.

They used range-based partitioning instead of hash-based partitioning for the partition key post_id.

This is because range-based partitioning on a monotonically increasing field such as post_id automatically ensures the partitions are created based on distinct time periods.

Since most read requests at Reddit target recent posts, it allows the Postgres engine to cache the indexes of the most recent partitions in its buffer pool. In other words, less Disk I/O and better performance.

So - what do you think about Reddit’s solution to the metadata problem?

Would you have done something differently?

Leave a comment

References:

• The Reddit Media Metadata Store

• pgBouncer

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

As you might know, Agoda’s goal is to aggregate and provide the best prices from various external suppliers (hotels, restaurants, cab providers, etc). To ensure that customers always get the most up-to-date and accurate pricing information, Agoda’s supply system needs to process a massive amount of price updates from the suppliers in real time.

For example, a single supplier can provide 1.5 million price updates and offer details in just one minute. Any delays or failures in reflecting these updates can lead to incorrect pricing and booking failures.

Here’s how the typical Kafka application setup within Agoda’s supply system looks like:

The key components are as follows:

• Distributor: The component consumes price updates from various 3rd party systems (suppliers) and distributes processing jobs to the downstream processor service across multiple data centers.

• Processor: The processor service is responsible for transforming, processing, and delivering the updates to the downstream services and databases.

• Kafka: It acts as the messaging backbone between the distributor and processor components.

One of the key features of Kafka that makes it suitable for such large-scale data processing is the use of partitions.

Partitions help Kafka achieve parallelism by distributing the workload across multiple partitions and consumers. In other words, it can process messages faster and more efficiently.

However, Agoda’s online travel booking platform had some challenges that were made more apparent by Kafka’s parallelism.

Partitioner and Assignor Strategy

The partitioner and assignor strategies in Kafka play a crucial role in determining how messages (price updates from suppliers) are distributed across partitions and consumers.

• Partitioner: Determines how messages are distributed across partitions when they are produced. For example, round-robin strategy and sticky partitioning.

• Assignor: Determines how partitions are assigned to consumers within a consumer group. For example, range assigner and round robin assigner.

See the diagram below for a round-robin partitioner and round-robin assignor strategy:

Traditionally, these strategies were designed to work under the assumptions of homogenous consumer processing capabilities and uniform message workloads.

However, in Agoda’s case, these assumptions didn’t hold true leading to load-balancing challenges related to Kafka.

Challenges

The two main challenges are as follows:

• Heterogenous Hardware: Agoda uses a private cloud with Kubernetes which can lead to pods running on servers of different hardware generations. Benchmarks show significant performance differences between hardware generations.

• Uneven Workload: Messages may require different processing steps, leading to varying processing times. For example, processing certain types of messages may require calling 3rd party APIs and executing database queries, resulting in latency fluctuations.

These challenges led to the over-provisioning problem.

The Over-provisioning Problem At Agoda

Agoda initially used the Kafka round-robin partitioner and assigner to distribute the same number of messages to each partition and pod.

However, this led to over-provisioning.

Over-provisioning involves allocating more resources than necessary to handle the expected peak workload efficiently.

In Agoda’s case, the reason was an inefficient distribution of the load across their Kafka consumers.

For example, consider an example where Agoda deploys Kafka consumers on heterogeneous hardware in their processor service. Suppose they have:

• 2 faster workers with a processing rate of 20 messages/second each.

• 1 slower worker with a processing rate of 10 messages/second.

In this setup, the total expected capacity would be 20 + 20 + 10 = 50 messages/second.

However, with a round-robin distribution, each worker would receive 1/3 of the overall messages regardless of processing speed. If the incoming traffic consistently hits 50 messages/second, here’s what would happen:

• The 2 faster workers can easily handle their share of ~ 16.7 messages per second each.

• The slower worker cannot keep up with its allocated ~16.7 messages per second, causing the lag to build up over time.

The diagram below shows this scenario:

To avoid high latency, Agoda would have to add extra resources to this setup to maintain processing SLAs. In this example, to process 50 messages/second, they would need to scale out to 5 machines. In other words, they end up over-provisioning by 2 extra machines due to the suboptimal distribution logic that doesn’t account for heterogeneous hardware.

Also, the same scenario can occur when the workload for each message is not identical. Either way, it results in:

• Increased hardware costs

• Wasted resources as some consumers sit idle while others are overburdened

• More maintenance overhead

The Solution Agoda Didn’t Adopt

The first obvious solution to overcome the over-provisioning challenge was static balancing. There were mainly two approaches in this regard:

• Deployment on identical pods

• Weighted load balancing

Deployment on Identical Pods

This solution involves controlling the types of hardware used in service deployments. However, it had some drawbacks:

• Maintaining a homogeneous hardware environment can be expensive, especially in a private cloud setup.

• Upgrading all existing hardware simultaneously can be challenging.

Due to these reasons, it was not adopted.

Weighted Load Balancing

In this approach, varying weights are assigned to different consumers based on processing capacity. Consumers with higher weights are routed more traffic.

This solution works well when:

• The capacity of consumers is predictable and remains static.

• The workload of messages is uniform, making it easier to estimate machine capacity.

However, there were some challenges in implementing this in a real production environment at Agoda:

• Messages often had non-uniform workloads, making capacity estimation difficult.

• Dependencies like network and 3rd party connections were unstable, leading to fluctuating capacities.

• Frequent addition of new features required extra maintenance to keep the weights updated.

Agoda’s Dynamic Lag-Aware Solution

Instead of the static rebalancing approaches, Agoda adopted a dynamic, lag-aware approach to solve the Kafka load balancing challenges. They implemented two main strategies:

• Lag-aware Producer

• Lag-aware Consumer

Lag-aware Producer

A lag-aware producer dynamically adjusts the partitioning of messages based on the current lag information of the target topic. Here’s how it works:

• The producer maintains an internal cache of partition lags to reduce the number of calls to Kafka brokers.

• Using the lag data, the producer applies a custom algorithm to publish fewer messages to partitions with high lag and more messages to partitions with low lag.

• When lags are balanced and stable, this approach guarantees an even distribution of messages.

For example, let’s consider a scenario where Agoda’s supply system has an internal producer that publishes task messages to its processor. The system has 6 partitions with the following lags:

• Partition 1 → 100 messages

• Partition 2 → 120 messages

• Partition 3 → 90 messages

• Partition 4 → 300 messages

• Partition 5 → 110 messages

• Partition 6 → 280 messages

In this case, the lag-aware producer would publish fewer messages to partitions 4 and 6 due to their high lag while publishing more messages to the other partitions with lower lag.

The diagram below illustrates this.

Lag-aware Consumer

Lag-aware consumers are used when multiple consumer groups subscribe to the same topic. Here’s how they work:

• In a downstream service like the Processor, a consumer instance experiencing high lag can proactively unsubscribe from the topic to trigger a rebalance.

• During the rebalance, a customized Assigner redistributes the partitions across all consumer instances based on their current lag and processing capacity.

• Kafka 2.4’s incremental cooperative rebalance protocol minimizes the performance impact of rebalancing, allowing for more frequent adjustments.

For example, consider a scenario where Agoda’s Processor service has 3 consumer instances consuming from 6 partitions:

• Worker 1 → Partitions 1 and 2

• Worker 2 → Partitions 3 and 4

• Worker 3 → Partitions 5 and 6

If Worker 3 is running on slower hardware, it may experience higher lags in partitions 5 and 6. In this case, Worker 3 can proactively unsubscribe from the topic, triggering a rebalance. The custom Assigner then redistributes the partitions based on the current lag and processing capacity of each worker.

Algorithms for Lag-aware Producer

Coming back to the Lag-aware producer, Agoda used two main algorithms:

• Same-Queue Length Algorithm

• Outlier Detection Algorithm

1 - Same-Queue Length Algorithm

The Same-Queue Length Algorithm aims to achieve equal queue lengths across all partitions by adjusting the number of messages published to each partition based on the current lag.

For example, consider a scenario with 4 partitions having the following lags:

• Partition 1 → 100 messages

• Partition 2 → 150 messages

• Partition 3 → 120 messages

• Partition 4 → 200 messages

The target queue length would be 200 (the maximum lag). The algorithm would then publish messages as follows:

• Partition 1 → Publish 100 messages (200 - 100)

• Partition 2 → Publish 50 messages (200-150)

• Partition 3 → Publish 80 messages (200 - 120)

• Partition 4 → Publish 0 messages (already at the target)

See the diagram below:

2 - Outlier Detection Algorithm

The Outlier Detection Algorithm identifies partitions with abnormally high lag (outliers) and temporarily stops publishing messages to those partitions to allow them to catch up.

It uses statistical methods like Interquartile Range (IQR) or Standard Deviation (STD) to identify outlier partitions.

For example, consider a scenario with 6 partitions having the following lags:

• Partition 1 → 100 messages

• Partition 2 → 120 messages

• Partition 3 → 300 messages

Using the IQR method, partition 3 is identified as an outlier (slow partition). The algorithm would then:

• Stop publishing messages to partition 3

• Evenly distribute messages among partitions 1 and 2.

• If partition 3 improves and is no longer an outlier, gradually resume publishing messages to them.

So - what do you think about Agoda’s load-balancing solution for Kafka? Would you have done something differently?

Leave a comment

Reference:

How We Solve Load Balancing Challenges in Apache Kafka

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

Let’s say this is the query:

SELECT name, age FROM users WHERE city = ‘New York’;

Here’s a diagram that shows the process visually:

Let’s go through each step in more detail:

STEP 1

When you hit run, the query string first reaches the Transport Subsystem of the database.

As the name suggests, this subsystem manages the connection with the client.

It also performs initial authentication and authorization checks, and if satisfied, lets the query go to the next step.

STEP 2

The query now reaches the Query Processor, which has two main parts:

• Query Parser

• Query Optimizer

The Query Parser breaks down the SQL query into its parts (SELECT, FROM, WHERE, etc). It checks for any syntax errors and creates a parse tree.

Then, the Query Optimizer goes through the parse tree, checks for semantic errors (e.g. - if the ‘users’ table exists or not), and determines the most efficient way to execute the query.

As output, the optimizer produces an execution plan.

STEP 3

Next, the execution plan makes its way to the Execution Engine.

The plan is made up of all the steps that are needed to execute the query.

The Execution Engine takes this plan and coordinates the execution of each step by calling the Storage Engine. It also collects the results from each step and returns a unified response to the upper layer.

STEP 4

The Execution Engine sends low-level read/write requests to the Storage Engine depending on the execution plan.

All of this is facilitated by the various employees of the Storage Engine.

For example,

• The transaction manager ensures that the query runs within a transaction for consistency.

• The lock manager acquires necessary locks on the “users” table to prevent conflicts.

• The buffer manager checks if the required data pages are already in memory. If not, it requests the data to be loaded from the disk into memory.

• The recovery manager logs the operations for rollback or recovery.

👉 So - what else would you add to this journey of an SQL query?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

Why?

Because your database is struggling to keep up.

This is a common problem in backend development—your code executes in milliseconds, but your database queries take seconds. If you don’t address this, you’ll end up with sluggish APIs, frustrated users, and a system that crumbles under load.

Let’s explore why databases slow down and how to fix them.

Why the Database Can Become a Bottleneck?

Databases are slower than code execution.

Your application code runs in memory, but your database reads and writes data from disks or over a network. Even with SSDs and optimized database engines, disk access and network calls are still much slower than in-memory operations.

But how do you know if your database is causing performance issues?

Look for these symptoms:

• Your API requests take a long time despite minimal application logic.

• Your database CPU usage is always high.

• Queries take longer as your data grows.

• Your application struggles under high user traffic.

Diagnosing the Problem

Before you start optimizing, you need to identify the actual bottlenecks. Here’s how:

Use Query Profiling

Every major database has a way to analyze slow queries:

• MySQL & PostgreSQL: Use EXPLAIN ANALYZE to check execution plans.

• MongoDB: Use .explain() to analyze queries.

• Slow Query Logs: Identify the queries that are taking the most time.

For example:

EXPLAIN ANALYZE SELECT * FROM orders WHERE status = 'pending';

This will show how your database processes the query and where the delays are happening.

Look at Key Performance Metrics

Monitor the following database metrics:

• Query execution time: Are some queries consistently slow?

• Read/write latency: Is your database struggling with high read or write operations?

• Locking issues: Are queries blocking each other?

• Connection pooling: Are too many connections being opened and closed?

Once you’ve identified the slow queries, you can start optimizing them.

Database Locking & Deadlocks

When multiple queries or transactions run concurrently, databases use locking mechanisms to ensure data consistency.

However, poor transaction management can lead to deadlocks, where two or more queries wait indefinitely for each other to release a lock, causing performance bottlenecks.

There are different types of locks:

• Row-level locks: Lock individual rows, allowing higher concurrency. (Ideal for OLTP applications)

• Table-level locks: Lock the entire table, preventing modifications while a transaction is running. (Can cause slowdowns)

• Shared vs. Exclusive locks: Shared locks allow multiple reads, while exclusive locks block all access until completion.

Some best practices to deal with deadlocks are as follows:

• Keep transactions short and efficient (commit early).

• Access tables in a consistent order across transactions.

• Use indexing to reduce unnecessary locks.

• Monitor for deadlocks using SHOW ENGINE INNODB STATUS; (MySQL) or pg_stat_activity (PostgreSQL).

Optimizing Your Queries

Optimizing queries is a quick win. Here are some common tips that can help:

1 - Add Indexes (The Right Way!)

Indexes speed up lookups by avoiding full table scans.

For example, the query below would be considered bad without indexes in the email field.

SELECT * FROM users WHERE email = 'user@example.com';

A simple fix would be to create the index on email.

CREATE INDEX idx_email ON users(email);

With an index, the database can locate the data instantly instead of scanning every row. However, too many indexes slow down writes, so there is a trade-off involved. Make sure to only index the columns you frequently search on.

2 - Avoid SELECT *** (Be Specific)

Fetching all columns when you only need a few wastes resources. For example, the below query should not be used:

SELECT * FROM orders WHERE status = 'pending';

A better option is the query below:

SELECT id, customer_id, total_price FROM orders WHERE status = 'pending';

This reduces the amount of data transferred and speeds up queries.

3 - Use Joins & Reduce Nested Queries

Avoid nested queries. For example:

SELECT * FROM orders WHERE customer_id IN (SELECT id FROM customers WHERE city = 'New York');

Using joins like the example below can be a much better strategy.

SELECT orders.order_id 
FROM orders 
JOIN customers ON orders.customer_id = customers.id 
WHERE customers.city = 'New York';

Joins are optimized by the database, whereas nested queries can cause multiple redundant lookups.

Scaling Beyond Queries: Architectural Optimizations

Some optimizations are also required on the architectural level to improve database performance. Here are some examples:

Schema Design

Database schema design directly impacts query speed, scalability, and storage efficiency. Poor schema decisions lead to slow queries, high storage costs, and scaling bottlenecks.

Here’s how to optimize schema design for better performance:

1 - Normalization vs. Denormalization: Trade-offs

• Normalization (3NF and above) reduces data redundancy by breaking tables into smaller, related entities.

  • Pros: Saves space and avoids update anomalies.

  • Cons: Can slow down read-heavy queries due to excessive JOINs.

• Denormalization merges frequently used data to reduce JOINs, making reads faster at the cost of extra storage and potential update issues.

  • Pros: Great for analytics & high-read workloads.

  • Cons: Can lead to data inconsistency if not managed properly.

Rule of Thumb: Normalize when you expect frequent writes, denormalize when reads dominate.

2 - Choosing the Right Data Types

• Use BIGINT instead of UUID for primary keys unless uniqueness across distributed systems is needed.

• Use BOOLEAN for true/false values.

• Use TEXT/JSONB sparingly. This is because indexable columns are faster for queries.

Use Caching

Instead of hitting the database every time, store frequently accessed data in a cache (Redis, Memcached). This reduces database load significantly.

For example, cache the result of expensive queries.

cache.set("user_123_orders", query_result, expire=300) # Expires in 5 minutes

Optimize Database Connections

Opening and closing database connections is expensive. Use connection pooling to reuse existing connections instead of creating new ones for every request.

For example, use pgbouncer for PostgreSQL connection pooling.

Materialized Views for Precomputed Expensive Queries

A Materialized View is a precomputed query result stored as a physical table. Unlike a regular database view, which runs a query every time it’s accessed, a materialized view stores the result persistently, making expensive queries much faster.

For example, suppose we frequently calculate total sales per day from a huge orders table:

SELECT order_date, SUM(total_price) AS daily_sales
FROM orders
GROUP BY order_date;

If the orders table is massive, this query slows down dashboards and reports. To handle this, you can create a materialized view. This way queries can read directly from the view instead of querying the orders table.

However, materialized views don’t usually auto-update when data changes. There needs to be a mechanism to update them. Also, this approach is not suitable for real-time data needs.

Sharding & Partitioning for Large Datasets

When your database gets huge, split data across multiple servers (sharding) or distribute tables across partitions.

For example:

• Shard users based on country (users_us, users_eu)

• Partition orders by date (orders_2023, orders_2024)

This reduces the load on a single table and speeds up lookups.

Use Read Replicas for Scaling Reads

If you have millions of reads per second, create read replicas to distribute the load.

For example:

• Primary DB: Handles writes and critical reads.

• Read Replicas: Handle read queries.

Many large-scale systems use this approach for redundancy as well as improved durability.

Conclusion: Diagnose Before You Optimize

Here are some tips to keep in mind:

• Don’t blindly optimize—use logs and profiling to find real bottlenecks.

• Start with query optimization—indexes, joins, and batching solve 80% of problems.

• Use caching and read replicas—offload repetitive queries.

• Think about scaling architecture—sharding, partitioning, and hybrid databases.

A slow database doesn’t have to hold you back. With the right optimizations, you can make your system blazing fast and scalable.

👉 So - which other tip will you add to optimize the database access?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

This monolith was responsible for both client and server-side functionality. The model, view, and controller layers were packed into a single repository.

A simple diagram for this architecture would appear as follows:

There were several advantages to this monolithic approach:

• Easy to get started initially.

• Good for agile development

• Manageable complexity levels

Those were simpler times and such times usually don’t last. Airbnb grew fast, and its engineering team doubled and tripled. Suddenly, you had more and more developers adding new code and features to the monorail.

Just like too many cooks spoil the dish, the code base started to get tightly coupled, and data ownership became muddled. Any developer could make changes to any part of the application, making it difficult to track and coordinate the changes.

Airbnb decided to migrate from the monorail to a Service-Oriented Architecture. You could also see it as a move towards loosely coupled microservices.

The Rules of the Game

Of course, Airbnb was wise and came up with some key design principles for building services. Some of the rules are mentioned below:

• A service should own both the reads and writes to its data. This is similar to the database-per-service pattern where a particular piece of data is owned by one and only one service.

• A service should address a specific concern

• Services should avoid duplicating functionality

• Data changes should happen via standard events to encourage loose coupling.

• Each service must have proper alerting and observability settings

The SOA Design

After a few years of migration stuff, Airbnb ended up with the below architecture:

Let’s unpack it a bit.

• The bottom layer is composed of data services. These services are the entry point for all interactions with the database and aren’t dependent on any other service.

• The derived services read data from other services and apply basic business logic. They can also have specialized data storage to store any derived data.

• The middle-tier services contain the key business logic that doesn’t fit at the data service level or the derived data service level.

• At the very top are the presentation services. These services aggregate data from all other services and apply frontend-specific business logic before returning the data to the client.

The Migration

The move from the monorail to shiny new services wasn’t an overnight process. Also, migrating reads and writes had their problems.

Let’s look at both scenarios and how Airbnb handled the migration.

Migrating the Reads

Airbnb used dual reads and response comparison for the old read path (through the monorail) and the new read path (the service).

See the diagram below:

Every read request went through both paths. The responses were sent to a comparison framework. Once the engineers were sure that things were as expected, the traffic to the service path was ramped up.

Migrating the Writes

Writes were trickier because one cannot dual-write to the same database. Therefore, the engineers used a shadow database.

See the diagram below:

The new service initially writes to the shadow database. After that, read requests are sent to the prod and shadow database, and the responses are sent to the comparison framework.

Only after the comparison came out clean, the writes were completely switched to the new service.

Lessons

Airbnb had several important lessons from this whole migration process. A few important ones for us are as follows

• Invest in common infrastructure pieces early on. Things like shadow databases, comparison frameworks, etc. can help during the migration.

• Simplify the service dependencies

• Moving to services is not just technical but also a cultural change for the organization.

• The migration isn’t a fixed destination but a journey.

So - what do you think about Airbnb’s migration journey? Would you have done something differently?

Leave a comment

References:

• Airbnb’s Migration from Monolith to Services

• Airbnb at Scale

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

Eraser, the developer-focused whiteboard tool, makes this extremely easy, and now it can do a lot more with new integrations. The premise of this new integration is amazing: instead of dragging and dropping UI elements, what if you could just tell your IDE to “draw the architecture diagram”?

Last week, I got to try the new AI integrations of Eraser and here’s what I learned:

The Setup

The biggest friction point in documentation is context switching. You usually have to leave your code editor, open a browser, log in to a tool, and start from scratch. However, Eraser makes this easy. All you need is install the skill in your IDE of choice by running a simple CLI command.

npx skills-add eraser-labs/eraser-io

With this command, we are essentially priming the IDE to work with Eraser. This allows the user to stay within an AI-powered editor (like Cursor) and generate diagrams without leaving their code.

How It Works

When we ask the AI agent to create a diagram using the Eraser skills, here’s what happens behind the scenes:

1. You describe the diagram you want in natural language

2. The AI agent processes your request:

   • Skills: The agent generates Eraser-specific syntax.

   • MCP Server: Can use the syntax or generate with natural language

3. Eraser returns a rendered diagram image link.

The diagram below shows this flow:

The agent can now generate any diagram type supported by Eraser:

• Architecture diagrams - Cloud infrastructure, system design

• Sequence diagrams - API flows, authentication sequences

• Flowcharts - Process flows, decision trees

• Entity relationship diagrams (ERD) - Database schemas

• BPMN diagrams - Business processes, swimlane diagrams

Trial 1: Generating OAuth Flow

To truly test a diagramming tool, you need a subject that is notoriously annoying to visualize. And therefore, I tried to visualize the OAuth flow:

If you have ever had to explain the OAuth flow to a junior developer or a stakeholder, you know the struggle. You have the User, the Client App, the Authorization Server, and the Resource Server. Then you add in the PKCE (Proof Key for Code Exchange) flow, and suddenly everything can start looking like a mess.

So, I tried it with Eraser this time, using a simple prompt. Instead of drawing the “Client” box, you simply prompt the agent: “Generate a sequence diagram for the Google OAuth2 flow using PKCE and OIDC discovery.”

Here’s what I got in just a few seconds:

Pretty amazing, if you ask me. And it also makes thing easy to build upon further if needed.

Trial 2: Visualizing Infrastructure (The Terraform Scenario)

Another trial I did was using the infrastructure diagrams that you can generate using Terraform files.

Imagine you join a new team and are handed a massive repository of Terraform files. You see a main.tf, variables.tf, and modules for VPCs, RDS instances, and EC2 auto-scaling groups. Reading the code tells you what is there, but it doesn’t tell you how it connects.

Using this AI integration, you could theoretically feed the agent your main.tf file and ask: “Read this Terraform configuration and generate an architecture diagram showing the relationship between the public subnets, the load balancers, and the database.”

The AI parses the HCL (HashiCorp Configuration Language), identifies the resources, understands the depends_on or reference links, and outputs a visual map.

This “Reverse Engineering” capability of Eraser is perhaps the strongest selling point for me. It transforms “documentation” from a proactive chore (something you have to create) into a reactive asset (something you can generate instantly from existing code).

What I Feel About The AI Integrations?

After seeing the capabilities, here is what I think about this skill and where it fits in a developer’s toolkit.

1. It Respects Flow State: The primary benefit isn’t just that the computer draws faster than you. It’s that you don’t have to stop coding to document. By keeping the generation inside the terminal/IDE, there is very little context switching. You stay in the terminal, you stay in the code, and the diagram happens in the background.

2. The “80/20” Rule of Diagramming: Is the AI-generated diagram perfect? Probably not 100% of the time. It might mislabel a specific edge case or group things awkwardly. However, it gets you 80% to 90% of the way there in seconds. It is much easier, in my view, to move a few pre-generated boxes around than to create a system design from a blank white screen.

3. Documentation as a Communication Tool: Often, backend developers build complex architectures that the Ops team has to deploy. Miscommunication here is common. Being able to instantly generate a visual representation of your code or Docker files bridges that gap.

Also, special thanks to the Eraser team for sharing the details about the new AI integrations and collaborating on this post.

You can try out these integrations yourself. Here’s the link to the documentation that can guide you through the process:

Eraser AI Integration Documentation

So, what do you think about these features?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

However, in a world where user attention spans are fleeting—averaging just 90 seconds before a decision is made—Netflix’s ability to deliver content swiftly is critical.

One of the core technologies that helps Netflix achieve this is EVCache, a distributed in-memory key-value store.

Here are four major use cases for EVCache at Netflix and how they contribute to keeping you streaming for as long as possible.

1 - Look-aside Cache

The first and perhaps most common use case for EVCache is as a look-aside cache.

This caching pattern ensures that frequently accessed data is readily available, reducing the time required to fetch data from backend services and databases.

How It Works:

• When an application requests data, it first queries the EVCache client to see if the data is already cached.

• If the data is not in the cache (a cache miss), the application fetches it from the backend service, which may involve querying the Cassandra database.

• Once the data is retrieved from the database, it is stored in EVCache for future requests.

Why It Matters:

• Reduced Latency: By serving data directly from the cache, Netflix can significantly reduce the time it takes to load recommendations, watch histories, or other user-specific data.

• Scalability: Backend services and databases are spared from handling repetitive queries, allowing them to focus on other tasks.

• Improved User Experience: Faster response times mean users can start streaming without delays, reducing the likelihood of them abandoning the platform.

Example Use Case:

Imagine a user navigating to the Netflix home page. The application needs to fetch personalized recommendations and thumbnails. Instead of querying the backend service for every user, EVCache serves these requests almost instantly, ensuring a smooth and engaging experience.

2 -Transient Data Store

Netflix also uses EVCache as a transient data store to manage temporary data that changes frequently but doesn’t require permanent storage. A prime example is playback session information.

How It Works:

• A playback session begins when a user starts watching a show or movie. This session information, such as playback time and device details, is stored in EVCache.

• Other services may update this session data (for example, when a user pauses or skips content).

• At the end of the session, the data may be discarded or stored in a more permanent database for analytics.

Why It Matters:

• Low Latency Updates: Transient data like session information requires real-time updates and reads, which EVCache handles efficiently.

• Service Coordination: Since multiple services interact with playback data during a session, EVCache ensures that this data is accessible to all relevant services with minimal latency.

• Cost Efficiency: EVCache provides a cost-effective solution for handling temporary data that doesn’t need to be stored long-term.

Example Use Case:

When a user pauses a movie on one device and resumes it on another, EVCache ensures that the playback position is instantly updated and available, enabling seamless transitions between devices.

3 - Primary Store

For some use cases, EVCache is not just a caching layer but acts as a primary data store. One prominent example is Netflix’s personalized home page, which is computed nightly for every user.

How It Works:

• Large-scale pre-compute systems analyze each user’s watch history, preferences, and recommendation algorithms.

• The result is a tailored home page, including categories, thumbnails, and recommendations for each profile.

• This data is written to the EVCache cluster, where it is stored and served to the user’s device during their next session.

Why It Matters:

• Pre-Computed Efficiency: By precomputing the home page, Netflix reduces the computational load during peak hours when users log in.

• Scalability: EVCache’s distributed nature allows Netflix to handle millions of unique home pages across its global user base.

• Instant Access: Storing the home page data in EVCache ensures low-latency retrieval, enabling Netflix to deliver a fully personalized experience as soon as the user opens the app.

Example Use Case:

When a user logs into their profile, the pre-computed data stored in EVCache is instantly retrieved to populate their home page. This allows Netflix to deliver recommendations without waiting for backend computations.

4 - High Volume Data

Netflix also uses EVCache for data that experiences high-volume access and requires high availability. One notable example is UI strings and translations shown on the home page.

How It Works:

• A separate process asynchronously computes and publishes UI strings, such as titles, descriptions, and menu text, to EVCache.

• These strings are then read by the application whenever needed, ensuring low latency and high availability.

Why It Matters:

• Global Reach: With users around the world, Netflix needs to deliver localized content and UI strings in multiple languages.

• High Availability: EVCache ensures that even under heavy traffic, essential UI elements are quickly and reliably available.

• Improved User Experience: By caching these high-volume data elements, Netflix ensures consistent and fast loading times across all devices and regions.

Example Use Case:

When a user opens Netflix, the localized UI elements, such as titles and navigation options, are fetched from EVCache. This ensures that the interface is fully loaded before the user starts interacting with the app.

👉 So - which other caching use case have you come across?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

Synchronous communication is when one service waits for a response from another before proceeding.

While it sometimes works well, relying solely on synchronous communication for every interaction between microservices is counter-productive.

The Challenges of Sync Communication

There are multiple challenges with just relying on synchronous communication.

Let’s look at a few important ones:

1 - Increased Latency

In a synchronous model, a service sends a request to another service and waits for a response.

If multiple services are chained in this manner, you just end up with latency compounding.

For instance, if Service A calls Service B, which then calls Service C, the overall response time is the sum of the latencies of each call. In scenarios where services are geographically distributed or under heavy load, this latency is bound to become a bottleneck.

2 - Downtime and Cascading Failures

When one service depends on another synchronously, any downtime or failure in one service can ripple through the entire system.

If Service B is unavailable, Service A will fail to process its requests. This tight coupling makes the entire system more fragile and prone to cascading failures.

3 - Higher Costs

Synchronous communication requires services to handle requests in real time.

To meet peak demand, you often need to provision resources for the highest possible load, leading to underutilized resources during off-peak times.

4 - Tight Temporal Coupling

Synchronous communication creates a dependency on the availability and responsiveness of other services at the instance level.

Both service instances must be operational simultaneously for communication to succeed. This temporal coupling can complicate deployments, scaling, and debugging.

Alternatives to Synchronous Communication

The good news is that there are alternative approaches to synchronous communication.

Let’s look at some of these alternatives:

Asynchronous Request-Response

In asynchronous communication, the caller sends a request but doesn’t block while waiting for the response. For example, a “Payment” service can place a “Process Payment” request in a queue. The “Billing” service processes the request asynchronously and places a “Payment Processed” message in the response queue.

A common pattern for implementing this is using message queues, such as RabbitMQ, Kafka, or AWS SQS.

How It Works:

• The sender places a request on a message queue.

• The receiver processes the request asynchronously and places the response back in the queue.

• The sender retrieves the response at its convenience.

Benefits:

• Non-blocking communication improves responsiveness.

• Services are loosely coupled, as they don’t need to be online at the same time.

• Message queues can act as a buffer, helping handle spikes in traffic.

Event-Driven Architecture

In an event-driven architecture, services communicate by publishing and subscribing to events. Instead of direct service-to-service calls, services emit events that other services consume.

For example, when an “OrderPlaced” event is emitted by the “Order” service, the “Inventory” service updates stock levels, and the “Notification” service sends a confirmation email—all without direct communication between them.

How It Works:

• Services publish events, such as “OrderPlaced” or “PaymentProcessed,” to an event broker (e.g., Kafka, AWS SNS, or Redis Streams).

• Interested services subscribe to these events and take appropriate actions.

Benefits:

• Decouples services, allowing them to operate independently.

• Enables scalability and resilience, as services don’t directly depend on each other’s availability.

• Simplifies adding new functionality, as new services can subscribe to existing events without modifying the publisher.

Well-Defined Service Boundaries

When services have well-defined responsibilities and minimal overlap, the need for inter-service communication is naturally reduced.

For example, instead of having a “User” service that handles user data and authentication, split these responsibilities into a “User Profile” service and an “Authentication” service. This minimizes unnecessary interactions and makes each service easier to scale and maintain.

How This Helps:

• By encapsulating functionality within a service boundary, you avoid frequent cross-service calls.

• Services can operate more independently, improving their reliability and scalability.

Bulk Data Transfers

For scenarios involving large amounts of data, consider using bulk data transfers instead of multiple synchronous requests. For example, a “Reporting” service can retrieve transaction logs from the “Payments” service in bulk, rather than querying for each transaction individually.

How It Works:

• Services periodically exchange data using batch jobs, file transfers, or database replication.

• These operations are performed asynchronously during off-peak hours.

Benefits:

• Reduces the load on services during real-time operations.

• Improves overall system efficiency.

Caching Local Copy

Sometimes, frequent inter-service communication can be replaced by caching or maintaining a local copy of data. For example, a “Product” service can cache product details locally instead of querying the “Catalog” service for every request.

How It Works:

• Use caching solutions like Redis or Memcached to store frequently accessed data.

• Replicate state across services to avoid constant data retrieval.

Benefits:

• Reduces latency by serving data locally.

• Minimizes the risk of service unavailability affecting performance.

Best Practices for Adopting Alternatives

• Understand the Use Case: Not every scenario requires asynchronous communication. Analyze the trade-offs and choose the approach that fits your requirements.

• Monitor and Debug: Use distributed tracing tools like Jaeger or Zipkin to monitor asynchronous flows and troubleshoot issues.

• Graceful Degradation: Implement fallbacks and retries to handle communication failures gracefully.

• Prioritize Idempotency: Ensure that operations are idempotent to handle duplicate messages in asynchronous systems.

👉 So - will you add any other alternative approach to synchronous communication?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

High latency can render an application effectively unusable, frustrating end-users and negatively impacting business outcomes just as much as a complete outage.

The frustration of a slow website is more catastrophic to modern-day users.

Considering this, it is wise for developers to invest in learning about low-latency strategies.

Here are the top strategies to reduce latency that you must be aware of (in no particular order):

1 - Caching

Caching is a fundamental technique used in distributed systems to improve performance and reduce the load on backend services.

The primary goal of caching is to minimize costly database lookups and avoid performing high-latency computations repeatedly.

By storing frequently accessed data in a cache, applications can retrieve the data much faster compared to fetching it from the source, such as a database or a remote service.

Caches are typically implemented using high-speed memory or fast storage systems that provide low-latency access to the cached data.

2 - Content Delivery Networks (CDN)

While caching is a powerful technique for improving performance and reducing the load on backend services, it can be taken to the next level by leveraging Content Delivery Networks (CDNs).

A CDN is a geographically distributed network of servers that work together to deliver content to end users with high availability and performance.

CDNs reduce latency by caching content closer to the end users.

• When a user requests content from a website or application that utilizes a CDN, the request is redirected to the nearest CDN server instead of the origin server.

• If the requested content is already cached on the CDN server, it is served directly from the cache, eliminating the need to retrieve it from the origin server. This reduces the distance the data has to travel and minimizes the response time.

3 - Load Balancing

Load Balancing is a super important technique to distribute workload between multiple machines.

By employing load balancers to distribute incoming traffic across multiple servers, we ensure that no single server becomes overwhelmed. In other words, the system performs according to the user’s expectations.

When a client sends a request to a distributed system, the load balancer acts as the initial point of contact. It receives the incoming request and intelligently routes it to one of the available servers based on predefined algorithms or policies.

Common load-balancing algorithms include:

• Round Robin

• Least Connections

• IP Hash

4 - Async Processing

When dealing with long-running tasks in a distributed system, it’s important to consider the user experience and the overall system responsiveness.

Implementing asynchronous processing is a powerful technique that allows the system to respond quickly to the user while performing further processing in the background.

Here’s how it works:

• In an asynchronous processing model, when a user initiates a long-running task, such as generating a complex report or processing a large dataset, the system immediately acknowledges the request and returns a response to the user.

• The response indicates that the request has been received and is being processed.

• Behind the scenes, the system enqueues the task into a message queue or a task queue.

• A separate background worker process or a pool of worker processes continuously monitors this queue, picking up tasks as they become available.

• The worker process executes the long-running task asynchronously, without blocking the main application thread or impacting the system’s responsiveness.

5 - Database Indexing

Databases can be a huge source of latency if not designed optimally.

To ensure optimal database performance and faster query processing, it is crucial to implement the right indexes and identify and refactor slow database queries.

Indexes play a vital role in improving the efficiency of database queries.

By creating appropriate indexes on frequently accessed columns or combinations of columns, the database can quickly locate and retrieve the required data without scanning the entire table. This reduces the time and resources needed to process queries, especially for large datasets.

6 - Data Compression

When transmitting data over a network, especially in distributed systems, it’s important to consider the size of the data being sent.

Larger data payloads can consume more network bandwidth, increase latency, and impact overall system performance.

To mitigate these issues, a common technique is to compress the data before sending it over the network.

Data compression involves reducing the size of the data by applying compression algorithms that remove redundancy and represent the data more efficiently. There are several benefits to compressing the data:

• Reduced bandwidth usage

• Improved latency

• Efficient resource utilization

• Cost savings

7 - Precaching

Pre-caching is a technique used to proactively cache data in anticipation of future requests.

The idea is to cache in advance data that has a high probability of being accessed so that you can serve it faster when requested by a user.

8 - Use Keep-Alive Connections

The TCP handshake is a critical process that establishes a reliable connection between two endpoints before data transmission can begin.

However, this process can be costly in terms of network latency and resource consumption, as it involves multiple round trips between the client and the server.

To optimize the overhead associated with establishing new TCP connections, it is recommended to reuse existing connections whenever possible.

Connection reuse, also known as connection pooling, is a technique that allows multiple requests to be sent over the same TCP connection, reducing the need to establish new connections for each request.

👉 So - which other low-latency techniques have you used?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

Sticky sessions make sure that all of a user’s requests are consistently directed to the same server instance for the duration of their session.

Here’s what it looks like:

But why is this important?

It’s important when the server instance physically stores some user-specific information without which the user’s experience gets impacted.

Here are a few examples:

• Session Objects: When a user establishes a connection with the server, the server may create a session object specific to that client-server pair. This object is stored in the server’s memory and contains information about the session.

• Shopping Cart Details: In an e-commerce application, a user’s shopping cart details may be stored on a specific server instance.

• File Processing: Consider a scenario where a user initiates a file processing task on a server. The server instance handling the task may need to maintain the state of the processing and provide updates to the user.

In all of these examples, only with sticky sessions, subsequent requests from the same user are directed to the same server instance so that the server can access and maintain the session object.

There are a few advantages to this approach:

• The user-specific data stored on a server remains consistent throughout the user’s session.

• You get better performance by directing a user’s requests to the same server instance.

• There’s also a cost advantage with sticky sessions where you don’t have to invest in external storage for session information.

Downsides of Sticky Sessions

Despite their advantages, however, sticky sessions have a couple of downsides:

• Sticky sessions will stick. Hence the name. They defeat the purpose of a load balancer because multiple users might get stuck to one instance while other instances sit idle and don’t receive any requests. Ultimately, this will create scalability challenges.

• The session is gone if the server goes down. Since the session is usually stored in memory, it’s tied to the life of the server instance. Of course, the user is redirected to another instance but the session information is gone. In practical terms, this might mean that the user needs to log in once more or stuff their shopping cart from scratch again. Not a great user experience, in my view.

Where Are Sticky Sessions Useful?

Sticky sessions can be useful in a couple of scenarios:

• WebSockets rely on persistent sticky connections.

• A basic implementation of the file-processing example where a user needs to check the status of a file being processed. Even there, for more advanced scenarios, you might want to store the status information for file processing in a separate store.

Alternative to Sticky Sessions

In my view, instead of relying on sticky sessions, it’s often better to store session information in a distributed session store.

A distributed session store is a separate, centralized storage system that holds session data. Popular options for distributed session stores include Redis and DynamoDB.

The diagram below shows this setup:

Here’s how it works:

• When a user’s request arrives at the load balancer, it can distribute the traffic evenly across the available server instances. No need for sticky sessions.

• The server instance that receives the request retrieves the relevant session information from the distributed session store.

• After processing the request, the server instance updates the session information in the distributed store. This ensures that the latest session data is persisted.

• Subsequent requests from the same user can be routed to any available server instance, as the latest session information can be retrieved from the distributed store.

Advantages of A Distributed Session Store

The key advantages of using a distributed session store are as follows:

• Fault Tolerance: If a particular server instance goes down, the session information is not lost.

• Scalability: As the system scales and more server instances are added, the load balancer can distribute traffic evenly without being constrained by sticky sessions. This is good for resource utilization and performance. Also, it helps make your application layer stateless.

• Flexibility: With session information in a distributed store, you have more flexibility in terms of server maintenance and updates. Server instances can be taken offline for maintenance without disrupting the user experience.

So - what do you think about the use of sticky sessions? Do you still prefer using them?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

In today’s post, we’ll look at the process of consistent hashing in a step-by-step manner so that it becomes easier to understand and explain.

Consistent Hashing is a technique used for distributing keys uniformly across a cluster of nodes.

The primary focus behind consistent hashing is to minimize the number of keys that need to be moved around when we add or remove a node from the cluster.

The below steps demonstrate how consistent hashing works:

STEP 1

The keys are hashed using a hash function.

The output range of these key values is treated as a fixed circular space or ring. For example, in the diagram below, K1, K2, K3, and K15 are the positions of the keys on the hash ring.

STEP 2

Next, the servers or nodes are also hashed using the IP address or the domain name as input.

We use the same hashing function to determine their respective positions on the ring.

See the below diagram:

STEP 3

Lastly, for every key, we traverse the ring in a clockwise direction starting from the position of the key.

Once a node is found, we store the key on that node.

See the below diagram:

Virtual Nodes in Consistent Hashing

Virtual nodes (often abbreviated as “v-nodes”) are a technique used in consistent hashing to address load balancing and uneven data distribution.

Instead of mapping a single physical node to one position on the hash ring, each physical node is mapped to multiple positions, creating “virtual nodes”. These positions are determined by applying a hash function multiple times with different input variations (e.g., appending a unique identifier like NodeID-1, NodeID-2, etc.).

For example, each physical node has 3 virtual nodes. After hashing, the virtual nodes might be distributed on the ring as follows:

• Node A: Virtual nodes A1, A2, A3

• Node B: Virtual nodes B1, B2, B3

• Node C: Virtual nodes C1, C2, C3

Keys (data) are hashed onto the ring and assigned to the first virtual node encountered while moving clockwise.

• Key K1 might fall near B2, so it is assigned to Node B.

• Key K2 might fall near A3, so it is assigned to Node A.

Since there are more positions (virtual nodes) on the ring, the data distribution is more evenly spread across physical nodes. Even if one key hashes poorly to a specific ring area, the effect is diluted across the multiple virtual nodes for each physical node.

On the downside, virtual nodes increase the size of the metadata the system needs to track, as multiple entries represent each physical node.

Advantages of Consistent Hashing

There are multiple advantages of consistent hashing:

• Scalability: Adding or removing nodes in the system only affects a small portion of the data, rather than requiring all data to be reassigned. This makes it ideal for distributed systems with dynamic membership.

• Load Balancing: By using techniques like virtual nodes (assigning multiple positions for each physical node), consistent hashing ensures a more balanced data distribution across servers.

• Fault Tolerance: When a node fails, only its keys need to be redistributed to other nodes. The rest of the system continues to function smoothly.

Disadvantages of Consistent Hashing

Consistent hashing also has some disadvantages that should be kept in mind:

• Imbalance Without Virtual Nodes: Without virtual nodes, the natural distribution of keys can be skewed, leading to uneven loads across nodes.

• Dependency on Hash Function: The quality of the hash function directly affects the balance of the distribution. A poorly chosen hash function can lead to clustering or inefficiencies.

Use Cases

Consistent hashing is widely used in various solutions. Here are a few important examples:

• Distributed Databases: Ensures even data distribution and minimal disruption when nodes are added or removed (e.g., Amazon DynamoDB).

• Caching Systems: Balances requests across cache servers and minimizes cache misses after server changes (e.g., Memcached).

• Content Delivery Networks (CDNs): Balances file storage and retrieval across geographically distributed servers.

👉 So - have you used Consistent Hashing?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

And how should you make the right choice in your project?

Let’s clarify it in today’s post.

Polling: A Resource-Intensive Approach

When ordering food at a restaurant, it would be impractical and embarrassing to constantly ask the waiter if your order is ready.

This scenario illustrates the inefficiency of polling, a method where a client repeatedly requests data from a server at predefined intervals.

Polling involves one service making repeated requests to another service to check for new data.

Here are the key characteristics of polling:

• Repeated Requests: In polling, the client sends requests to the server at regular intervals, asking if there is any new data available. This approach is akin to constantly asking, “Do you have something new for me?” even when there might be no updates.

• Resource-Intensive: Polling is resource-intensive and inefficient. It consumes network bandwidth and server resources, even when there are no updates available.

• Missed Real-Time Updates: Polling can lead to missed real-time updates. Since the client checks for updates at predefined intervals, it may not receive notifications immediately when data becomes available.

Webhooks: A Notification-Based Approach

Webhooks offer a more efficient and real-time alternative to polling. Here’s how webhooks work:

1. Callback URL: With webhooks, you set up a callback URL that one system uses to notify another system when new data is available. This approach is like having a built-in notification system where the server says, “I’ll let you know when something happens.”

2. Push-Based Data Delivery: Webhooks push data to the recipient service as soon as it becomes available, ensuring real-time updates. This makes webhooks ideal for scenarios where timely notifications are crucial.

3. Efficiency and Real-Time Updates: Webhooks are more efficient than polling because they eliminate the need for repeated requests. They provide real-time updates, making them perfect for applications like payment notifications, CI/CD platforms, and other services that require immediate updates.

When to Use Polling and Webhooks

Each approach has its use cases:

1. Polling:

   • Frequent Data Updates: Polling is better when you don’t need real-time data updates and the data is frequently updated. Sending webhooks for every event can be resource-intensive. For example, a social media app that wants to report on its user base might be overwhelmed with webhooks if many users sign up in quick succession.

   • Flexibility: You can customize the sync frequency based on how often you need updated data.

2. Webhooks:

   • Real-Time Updates: Webhooks are perfect for scenarios where real-time updates matter. Examples include payment notifications, chat applications, and any service that requires immediate notifications.

   • Efficient Resource Utilization: Webhooks are more efficient in terms of resource utilization compared to polling. They ensure that data is delivered in real time without the need for repeated requests.

So - have you used Polling or Webhooks in your application?

Leave a comment

Shoutout

Here are some interesting articles I’ve read recently:

• Read this if you want a successful year at work in 2026 by

• How to Ship Fast Without Creating a Mess by

• The Systematic AI Code Review Workflow: Plan, Generate, Validate by

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

It’s one of the few techniques in software development that can be applied at every layer of your application.

Things start right from the front-end layer and end at the database layer. The only thing that changes is the overall usage pattern of the cache.

The diagram below shows the utilization of cache across a typical application stack.

Let’s look at each stage in a little more detail.

1 - Front End App

Browsers cache stuff all the time - HTTP responses, asset files, HTML pages, and so on.

Developers can set the caching directives in response headers to control the caching behavior for specific assets.

2 - Content Delivery Networks

CDNs can cache static content like images, stylesheets, and JS files.

Their advantage is caching stuff closer to the users, reducing latency in the process.

3 - Load Balancers

Some load balancers can also cache frequently accessed data.

Responses can be served without calling the backend server. This improves the response times and also reduces the load on the server or database.

4 - Service Instance Cache

Service instances can employ their caching solutions to boost performance.

This can be in the form of in-memory caches that are checked before fetching from the database or even the distributed cache.

5 - Distributed Cache

A more robust caching solution can use distributed caches like Redis.

They help with faster reads and writes than going to a traditional relational database. Also, you can scale the cache across multiple nodes for greater resilience.

6 - Full-text Search Tools

Searching through tons of text information is a time-consuming task.

Tools like ElasticSearch help index the data for faster text search, effectively acting as a cache.

7 - Databases

You might think that databases always fetch from the disk.

But databases also use special caching mechanisms. For example, a Bufferpool is like a cache within the database that holds data pages to support faster reads.

Also, Materialized Views store results of pre-computed results of expensive queries for faster access

Lastly, many new-age databases have their built-in cache solutions.

So - for how many layers have you used caching?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

While computing has many applications even when storage is not involved, it’s hard to imagine the utility of software systems for a majority of normal users in the absence of storage systems like databases.

Naturally, this means that the performance and scalability of databases play a big role in shaping an application’s user experience.

There are so many factors that can impact the performance of a database. Some important ones are as follows:

• Item Size: The average payload size of an item stored in the database determines whether the workload is CPU-bound or storage-bound.

• Item Type: The type of item directly impacts the kind of compression that is possible. For example, if you’re storing text, you can take advantage of a high compression ratio. Not so much when you store images, videos, or encrypted data.

• Total Dataset Size: This one directly impacts the infrastructure options and whether you’d need to go for replication and sharding.

• Throughput and Latency Requirements: Just saying “high throughput” is not enough. It’s critical to know the target throughput for choosing the optimal database type and infrastructure.

Having said that, databases also evolve along with your application. From time to time, you may need to come up with new strategies to extract the best performance and scalability out of your database.

Let’s look at a few tried and tested strategies that can boost the performance of the database as well as help you scale it better.

1 - Indexing

In databases, an index acts as a “table of contents” for your data.

It allows the database to quickly locate and retrieve specific information without scanning through every single record. Without an index, the database would have to scan the entire table to find matching records.

For example, if you have a “Customers” table with columns like “ID”, “Name”, “Email”, and “City”. In case you often search for customers by email, creating an index on the “Email” column can make a huge difference.

Key benefits of indexing are:

• Faster queries

• Reduced resource usage

• Increased concurrency

The trade-offs are as follows:

• Extra disk space for each indexed column

• More work during write operations to update the index.

2 - Materialized Views

A materialized view is like a snapshot of a query result, stored separately from the original data.

It’s derived from one or more tables or views and is maintained independently. Think of it as a pre-calculated summary that you can quickly access whenever you need it.

This materialized view can be refreshed periodically to keep the data up to date.

Key benefits of materialized views are:

• No need to run complex and time-consuming queries in the user request flow.

• Reduced load

The trade-offs are as follows:

• Extra storage

• Higher refresh time and inconsistent data in the view

3 - Denormalization

Denormalization involves duplicating data across multiple tables to optimize query performance. The goal is to reduce the number of joins and computations required to retrieve data, making queries faster and more scalable.

In other words, the rules of normalization are relaxed a bit in favor of performance.

Imagine an e-commerce store with a “Customers” table and an “Orders” table. In a normalized design, the “Orders” table would only store a reference to the “Customers” table.

To get the customer details with the order information, you’d need to join the two tables. However, as the number of orders grows, the join operation can become a performance bottleneck.

That’s where denormalization comes in handy.

By storing a field like the “CustomerName” directly in the “Orders” table, you can retrieve the details along with order information in a single query. Note that this is just an example scenario and doing so may depend on the overall system context and the data modeling rules in place.

Benefits of Denormalization:

• Faster queries

• Reduced overhead

• Improved read performance

The trade-offs are as follows:

• Data redundancy.

• Complex updates since redundant data may need to be kept in sync across multiple tables.

• Potential inconsistency.

4 - Vertical Scaling

Vertical scaling, also known as “scaling up,” is a technique that focuses on increasing the hardware resources of a single server.

It’s all about making your database server bigger, stronger, and faster.

A few easy ways to do so are:

• Upgrading to a faster CPU

• Adding more memory to your server

• Switching to high-performance storage devices like SSDs.

There are several benefits of Vertical Scaling:

• Better performance

• Simplified management

• Reduced latency

However, there are some limitations as well:

• There’s a limit to how much you can vertically scale a server before hitting some cost constraints.

• A single server failure can bring down your database.

• Upgrading hardware can be expensive.

5 - Caching

Caching is a technique that involves storing frequently accessed data in a high-speed storage layer, separate from the primary database.

When an application receives a request for data, it first checks the cache. If the data is found (a cache hit), it’s quickly retrieved without bothering the database. If the data is not found (a cache miss), the application fetches it from the database and stores a copy in the cache for future requests.

The benefits of Caching are as follows:

• Reduced database load

• Improved read performance

There are some trade-offs as well:

• Additional complexity

• Extra cost

• Chances of stale data

6 - Replication

Replication is a technique used in database systems to create and maintain multiple copies of data across different servers or nodes.

In a typical leader-follower replication model, one node is designated as the leader, while the others are followers.

The leader handles all the write operations ensuring consistency and integrity. The changes are automatically shared with the follower nodes whenever a write operation is performed on the leader node. The leader can also handle critical read operations where high consistency is required.

The follower nodes, in turn, handle read operations and help distribute the workload to improve performance.

The benefits of Replication are as follows:

• Improved Read Performance

• High Availability

• Durability

There are also some trade-offs:

• Introduces some delay in data synchronization known as replication lag

• Increased complexity

7 - Sharding

Database sharding is a technique that partitions a single large database into smaller, more manageable units called shards.

In a sharded database architecture, data is distributed across multiple shards based on a specific sharding key. The choice of the sharding key determines how the data is allocated to different shards.

Common sharding strategies include:

• Range-based Sharding: Data is partitioned based on a range of values of the sharding key.

• Hash-based Sharding: A hash function is applied to the sharding key to determine the target shard.

• Directory-based Sharding: A separate lookup table is maintained to map the sharding key to the corresponding shard.

The benefits of Sharding are as follows:

• Sharding allows for horizontal scaling of the database.

• Queries and write operations are processed in parallel.

• Reduced hardware costs when compared to vertical scaling.

However, there are also trade-offs:

• Sharding introduces additional complexity.

• Rebalancing data across shards can be a complex and time-consuming process.

• Joining data across shards can be challenging.

So - how do you improve the performance and scalability of the database?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another edition — Saurabh

Choose a database with a storage engine that follows the LSM Tree approach to store and retrieve data.

An LSM Tree, which stands for Log-Structured Merge-Tree, is a data structure designed to store and retrieve key-value pairs efficiently. It is specifically optimized for high write throughput.

Storage engines based on this data structure are called LSM storage engines. For example, LevelDB and RocksDB are a couple of key-value storage engine libraries that use LSM trees. Similar storage engines are also used in databases like Cassandra and HBase.

4 Steps to LSM Storage Engine

Here’s how a typical LSM Storage Engine works:

• STEP 1: When a write request for a key-value pair arrives, it is added to an in-memory structure known as memtable.

• STEP 2: When the memtable gets bigger than some threshold, it is flushed to disk as an immutable file called an SSTable (Sorted String Table). This is handled asynchronously so that the new requests are not blocked.

• STEP 3: To serve a read request, first try to find the key in the memtable, then in the most recent SSTable, then in the next older SSTable, and so on.

• STEP 4: To keep the number of SSTable files in check, run a periodic merging and compaction process in the background to combine them and discard overwritten or deleted values.

Components of an LSM Tree

What is an LSM Tree made up of?

It has a few key components:

• Memtable

• SSTable

• Compaction Process

The diagram below shows the overall design of LSM Trees

1 - Memtable

Imagine having a notebook where you write down new information as it comes in. In an LSM Tree, this notebook is called a memtable.

It’s a place in memory where all the new data is initially stored. You can use well-known tree data structures (such as red-black trees or AVL trees) to insert keys in any order and read them back in a sorted order.

For example: let’s say you are running a bookstore and want to keep track of your inventory. Whenever you receive a new shipment of books, you write down the book name and quantities in your memtable.

Here’s how the memtable looks like:

2 - SSTable

SSTables are files on disk that store the data from the memtable in a sorted order.

When the memtable reaches a certain size threshold, its contents are flushed to disk as an SSTable file.

For example: taking our bookstore example, here’s how the data might look in the SSTable

As more updates come in, new memtables are created and flushed to disk as additional SSTables.

3 - Compaction Process

Over time, you might end up with many SSTable files, making it harder to find specific information.

As the number of SSTables grows, the LSM Tree performs compaction to merge the smaller SSTables into larger ones. During compaction, the data is merged based on the key, and the latest values for each key are retained.

In our bookstore example, the data after compaction looks like this:

In this compacted SSTable, the latest values for each book are stored. For example, “Book A” has a quantity of 8, which is the most recent value from SSTable 2.

How LSM Trees Differ from B-trees?

Let’s look at the key differences between LSM Trees and B-Trees in some important aspects.

1 - Write Optimization

• LSM Trees: Optimized for high write throughput. As we discussed, writes are initially stored in a memtable and later flushed to disk, allowing for fast and efficient write operations.

• B-Trees: Writes require traversing the tree and potentially causing cascading splits or merges, which can be slower compared to LSM Trees.

2 - Read Performance

• LSM Trees: Reads may require accessing multiple levels of SSTables, resulting in higher latency compared to B-Trees. Techniques like Bloom Filters can help but we will discuss them later.

• B-Trees: Reads are faster as data is stored in a sorted manner, and the tree structure allows for efficient lookups.

3 - Space Amplification

• LSM Trees: Due to the multi-level structure and the need for compaction, LSM Trees may have higher space amplifications compared to B-Trees.

• B-Trees: They have lower space amplification as data is stored more compactly.

4 - Compaction Overhead

• LSM Trees: Compaction is a background process that merges SSTables, which can consume I/O and CPU resources.

• B-Trees: There is no separate compaction process, but they may require rebalancing and merging operations.

👉 So - have you used databases that support LSM Trees?

Leave a comment

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share

See you later with another value-packed edition — Saurabh

Even in the best of times, you might end up in a mess.

But if you’ve to build microservices for some reason, it’s better to know some patterns that can make your life easy.

Here are 4 must-know patterns for building microservices:

Database Per Service

As the name suggests, this pattern proposes that each microservice manages its own data.

This implies that no other microservice can directly access or manipulate the data managed by another microservice.

Any exchange or manipulation of data can be done only by using a set of well-defined APIs.

The figure below shows an example of a database-per-service pattern.

At face value, this pattern seems quite simple. You can implement it relatively easily when you are starting a brand-new application.

However, when you are trying to migrate an existing monolithic application to a microservices architecture, the demarcation between services is not so clear.

Most of the functionality is written in a way where different parts of the system access data from other parts informally.

Two main areas that you must focus on when using a database-per-service pattern are:

• Defining bounded contexts for each service.

• Managing business transactions spanning multiple microservices.

Shared Database

The next important pattern is the shared database pattern.

Depending on how you are looking at things, this might also sound like an anti-pattern.

That’s because this pattern adopts a much more lenient approach by using a shared database accessible to multiple microservices.

Here’s what it looks like in practice:

For existing applications transitioning to a microservices architecture, this is a much safer pattern as we can slowly evolve the application layer without changing the database design.

However, this approach takes away some key benefits of microservices.

• Developers across teams need to coordinate schema changes to tables

• Runtime conflicts may arise when multiple services are trying to access the same database resources.

Thanks for reading System Design Codex! Subscribe for free to receive new posts and support my work.

API Composition

The API Composition pattern attempts to solve the problem of implementing complex queries in a microservices architecture.

An API composer invokes other services in the required order. After fetching the results, it performs an in-memory join of the data before returning it to the caller.

However, this is an inefficient approach due to in-memory joins on potentially large datasets.

Check the illustration below:

CQRS and Event Sourcing

CQRS stands for Command Query Responsibility Segregation.

In the CQRS pattern, an application listens to domain events from other microservices and updates a separate database for supporting queries and views.

You can then serve complex aggregation queries from this separate database while optimizing the performance and scaling it out as needed.

Event Sourcing takes it a bit further by storing the state of the entity or the aggregate as a sequence of events.

Whenever there is an update or an insert on an object, a new event is created and stored in the event store.

You can use CQRS and Event Sourcing together to solve a lot of challenges around event handling and maintaining separate query data. This way, you can scale the read and writes separately based on specific requirements.

On the downside, this is an unfamiliar style of building applications for many developers and there are also multiple moving parts to manage.

Check out the illustration below that shows CQRS and Event Sourcing together.

So - have you used any of these patterns?

Leave a comment

Shoutout

Here are some interesting articles I read this week:

• 2-Tier to 3-Tier Architecture: Migration Journey With Modular Monolith and GraphQL by

• Essential Skills for Engineers to Thrive in the AI Era by

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

Share