3 Kafka Messaging Strategies

How to choose the right one?

Saurabh Dashora

Oct 15, 2024

“What should be the next step after publishing an event?”

This question came up during a design discussion for a new project.

There were two different points of view:

We should just forget about the event after publishing it to the broker.
We should make sure to get a confirmation that the broker received the event.

The problem, of course, is a game of trade-offs.

We were using Kafka for that project. And it turned out there were 3 options we could choose from:

Fire and Forget
Synchronous Send
Asynchronous Send

How Does the Kafka Producer Work?

Before we look at each option, let’s first understand how the Kafka Producer works.

In the first step, we create a ProducerRecord.

The record contains two mandatory items:

Kafka Topic (the place where we want to send the message)
The Message (what we want to actually send)

Once we create a ProducerRecord in our application code, we call the send method. This is where the Producer kicks into action and executes a bunch of stuff:

Serialize the message (key and value objects) to byte arrays to send the data over the network.
Choose a partitioner in case the sender hasn’t specified one. This is usually based on the key of the message.
Batch the records for a particular topic and partition.
Send the batch to the right Kafka broker.

Here’s what it looks like:

You can play around with the diagram on Eraser.io

So, what happens when the broker receives the message?

It sends back a response to the Producer. Think of it as an acknowledgment. More precisely, it’s the RecordMetadata object.

The object contains information such as topic, partition, and the offset of the record within the partition.

But that’s the happy path!

What happens if the broker fails to write the message for whatever reason?

In that case, the Producer gets an error and can retry sending the message a few more times before giving up and calling it a day.

So, what’s the deal with the 3 message-sending strategies?

Let’s find out.

Fire and Forget

In this strategy, we send a message to the Kafka broker and forget about it. We simply don’t care what happens to it.

Since Kafka is highly available, it will likely arrive on the other side successfully. In case of a minor issue, the Producer will retry sending the message automatically.

But yes, fire and forget means that messages can and will get lost. Also, the application won’t get any information or exceptions about these lost messages.

This is what it looks like in the code:

ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "kafka trial message");

try {
     producer.send(record);
} catch(Exception e) {
     e.printStackTrace();
}

Synchronous Send Approach

The strategy sounds dubious.

Why would someone want a messaging system to have synchronous behavior?

But Kafka lets you force a Producer to behave in a synchronous manner.

Here’s the code for the same:

ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "kafka trial message");

try {
     producer.send(record).get();
} catch(Exception e) {
     e.printStackTrace();
}

What’s the difference?

Nothing much!

We have simply turned the send method to a synchronous operation by calling producer.send(record).get().

The send() method returns a Future object. By using the get() method, we wait on the Future object to make sure if the call to send() was successful or not.

If not, the method throws an exception.

Asynchronous Send

This is the third approach.

Here you call the send() method of the Producer with a callback function. The callback function gets triggered when it receives a response from the Kafka broker.

ProducerRecord<String, String> record = new ProducerRecord<>("topic-1", "msg", "new kafka async");

try {
		producer.send(record, new DemoProducerCallback());
} catch(Exception e) {
		e.printStackTrace();
}

The DemoProducerCallback is the callback class. The send() method takes an instance of that class.

When to Use Which Strategy?

Here are a few pointers to consider:

👉 In my experience, Asynchronous Send is what you’d be using a majority of the time.

This strategy lets you send messages at high performance while also managing error situations and exceptions if any.

That’s what you want in a typical production system for any important domain requirement.

👉 Try to avoid Synchronous Send. It sounds safer, but you are essentially making a trade-off on performance.

Brokers in a typical Kafka cluster may take some time to respond to requests. With this strategy, you block the sending thread of your application. Not good for performance.

👉 What about Fire and Forget?

In my opinion, it has its uses. Since it is the best-performing in terms of throughput, you should use it for messages where delivery isn’t extremely critical.

Think of logs or sensor information where missing a few messages here and there isn’t a huge problem.

So - which strategy do you prefer? And how would you make the choice?

Shoutout

Here are some interesting articles I’ve read recently:

That’s it for today! ☀️

Enjoyed this issue of the newsletter?

Share with your friends and colleagues.

See you later with another edition — Saurabh

System Design Codex

Discussion about this post