Kafka Internals — Part 1
What really happens after producer.send()? A deep dive into the Kafka producer pipeline, batching, networking and broker communication.
Writing & Ideas
Technical explorations, engineering concepts and lessons learned while understanding how software systems work beneath the surface.
From Java internals to distributed systems, databases and the architecture behind reliable software.
23
Published articles
Latest writing
What really happens after producer.send()? A deep dive into the Kafka producer pipeline, batching, networking and broker communication.
Producer
Application sends a record
Serialization
Key and value are serialized
Record accumulator
Records are grouped into batches
Broker
Batches are sent to the cluster
The article collection
Browse articles by topic, from programming fundamentals to the internals of distributed systems.
Showing 1–6 of 23 articles
01 / 04
What really happens after producer.send()? A deep dive into the Kafka producer pipeline, batching, networking and broker communication.
Understanding how Kafka brokers coordinate, manage metadata, elect leaders and maintain a distributed cluster.
Building an understanding of topics, partitions, brokers, replication, offsets and consumer groups.
A deep dive into low-level design, object modelling, design decisions and engineering trade-offs.
Exploring the Strategy pattern through a practical business problem rather than an interview-only example.
Understanding dirty reads, non-repeatable reads and phantom reads through concurrent database transactions.
Continue the conversation
I write to understand things better, share what I learn and explore the engineering decisions behind software.