Kafka is a distributed event streaming platform. It provides a structure for publishing and subscribing to large volumes of events in real time. It serves real-time data pipelines, event-driven architectures, log aggregation, and more.

Topics and Partitions

Topics

In Kafka, messages are published to a Topic. A topic is a logical category of messages. Topics are created per event type: order-events, user-signups, and so on.

A topic is an append-only log that stores messages. Once written, messages are immutable. They are deleted when the retention period expires.

Partitions

A single topic is divided into multiple Partitions. Partitions are the core unit that provides both parallelism and ordering guarantees.

flowchart LR
    subgraph Topic["Topic: order-events"]
        P0["Partition 0
msg0, msg3, msg6..."]
        P1["Partition 1
msg1, msg4, msg7..."]
        P2["Partition 2
msg2, msg5, msg8..."]
    end

Messages within a partition maintain order. Across partitions, no ordering is guaranteed. Messages with the same key are assigned to the same partition, ensuring event ordering for a specific entity (e.g., a particular order).

Increasing partition count increases throughput, since multiple consumers can process each partition in parallel.

Offset

Each message within a partition has a unique Offset number, starting from 0 and incrementing sequentially. Offsets serve as the reference point for tracking “how far a consumer has read.”

Segments

A partition is a logical unit. On disk, it is split into smaller file units called Segments. A single partition spans multiple segments in sequence.

The broker appends messages only to the most recent segment, called the Active Segment. When the active segment reaches a set size (1GB by default, segment.bytes) or a set time elapses (segment.ms), the broker closes the current segment and opens a new one. This transition is called Rolling.

flowchart LR
    msg["new message"] --> S2
    subgraph Partition["Partition 0"]
        direction LR
        S0["Segment 1
base offset 0
(closed)"]
        S1["Segment 2
base offset 170410
(closed)"]
        S2["Segment 3
base offset 340820
(active)"]
    end

Each segment consists of three kinds of files.

  • .log: the file where actual messages are written
  • .index: an index mapping offsets to physical positions (bytes) within the file, used to locate a specific offset quickly
  • .timeindex: an index mapping timestamps to offsets, used for time-based lookups

A file is named after its segment’s base offset. 00000000000000000000.log is the segment starting at offset 0, and 00000000000000170410.log is the segment starting at offset 170410. When a consumer requests a specific offset, the broker decides which segment to open from the file name alone, then uses the .index to find the position within it.

flowchart TB
    subgraph Seg["Segment (base offset 170410)"]
        L["...170410.log
actual messages"]
        I["...170410.index
offset → byte position"]
        T["...170410.timeindex
timestamp → offset"]
    end

Segments are also the unit of retention. Kafka does not delete individual messages. It deletes whole segment files once their retention period expires (7 days by default, log.retention.hours). The active segment is never a deletion target, so a partition always retains at least its active segment.

Beyond deletion, there is another retention policy: Compaction (cleanup.policy=compact). It cleans up by key rather than by time, keeping only the latest value for each key and removing older ones. __consumer_offsets, which stores consumer offsets, is a representative compacted topic. Compaction is used when only the latest state per key matters (change history, state snapshots, and the like).

Kafka stays fast despite segments being disk files because of how they are stored. Messages are written only sequentially to the end of the active segment, so it leverages the disk’s sequential I/O performance directly. Recent data remains in the OS Page Cache, so most reads never reach the disk. When delivering to consumers, Zero-copy (sendfile) transfers data straight from the kernel buffer to the network, skipping a copy into application memory.

Producer

A Producer publishes messages to a topic.

When a producer sends a message, it must decide which partition to target.

Key-based partitioning. When a message has a key, a hash of the key determines the partition. The same key always maps to the same partition. This is used when event ordering for a specific user or order is required.

Round robin. Without a key, messages are distributed across partitions in sequence. Suitable when ordering is unnecessary and even load distribution is desired.

Custom partitioner. Custom partitioning logic can be implemented. Used when specific business rules dictate partition selection.

Acks

The producer can configure the level of acknowledgment required from brokers.

  • acks=0: No acknowledgment. Fastest, but messages can be lost.
  • acks=1: Leader broker acknowledges after writing. Messages can still be lost if the leader fails before replication.
  • acks=all: All ISR (In-Sync Replicas) acknowledge. Safest, but increases latency.

Delivery Semantics

The acks setting ultimately determines the message Delivery Semantics.

  • at-most-once: At most once. Loss is possible, but no duplicates.
  • at-least-once: At least once. No loss, but retransmission can produce duplicates. This is the Kafka producer’s default behavior.
  • exactly-once: Exactly once. Neither loss nor duplicates.

If a producer retransmits because it did not receive an acknowledgment, the same message can be written twice. The Idempotent Producer attaches a sequence number to each message so the broker filters out duplicate writes. In recent Kafka versions it is enabled by default, eliminating duplicates caused by retransmission.

Consumer

A Consumer reads messages from a topic. Unlike the producer’s “push,” consumers “pull” messages themselves, processing at their own pace.

Consumers commit the offset of messages they have read. Committed offsets are stored in an internal Kafka topic (__consumer_offsets). When a consumer restarts, it resumes from the last committed offset.

The gap between the offset a producer last wrote and the offset a consumer last committed is called Consumer Lag — the amount of messages not yet processed. Steady lag means the consumer keeps up with the incoming rate; continuously growing lag signals insufficient throughput.

Consumer Groups

Multiple consumers can be grouped into a Consumer Group. Within the same group, each partition is assigned to exactly one consumer.

flowchart LR
    subgraph Topic["Topic (3 Partitions)"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
    end
    subgraph Group["Consumer Group A"]
        C1["Consumer 1"]
        C2["Consumer 2"]
        C3["Consumer 3"]
    end
    P0 --> C1
    P1 --> C2
    P2 --> C3

If the number of consumers exceeds the number of partitions, the excess consumers remain idle. To increase throughput, increase the partition count first.

When consumers join or leave a group, Rebalancing occurs — the process of reassigning partitions. The early (eager) approach revoked all partition assignments during a rebalance, stopping the entire group. The later Cooperative Rebalancing reassigns only the affected partitions, letting the remaining consumers keep processing.

Multiple Consumer Groups

Different consumer groups read the same topic independently, each managing offsets separately.

flowchart LR
    subgraph Topic["Topic (3 Partitions)"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
    end
    subgraph GA["Group A (Order Processing)"]
        A1["Consumer A1"]
        A2["Consumer A2"]
    end
    subgraph GB["Group B (Analytics)"]
        B1["Consumer B1"]
    end
    P0 --> A1
    P1 --> A2
    P2 --> A1
    P0 --> B1
    P1 --> B1
    P2 --> B1

Multiple consumer groups subscribing to a single topic is the pub/sub pattern. A common example: an order processing system and an analytics system independently consuming the same events.

Brokers and Clusters

Broker

A Broker is a Kafka server instance. It receives messages, persists them to disk, and delivers them to consumers. Multiple brokers form a Cluster.

Each partition is assigned to one broker as the Leader. Producers and consumers communicate with the leader broker.

Replication

Partitions are replicated across multiple brokers. The number of brokers a partition is replicated to is called the Replication Factor. With a replication factor of 3, one leader and two followers sit on different brokers. If the leader fails, one of the followers is promoted to the new leader.

flowchart TB
    subgraph Cluster["Kafka Cluster"]
        subgraph B1["Broker 1"]
            P0L["P0 (Leader)"]
            P1F["P1 (Follower)"]
        end
        subgraph B2["Broker 2"]
            P0F["P0 (Follower)"]
            P1L["P1 (Leader)"]
        end
        subgraph B3["Broker 3"]
            P0F2["P0 (Follower)"]
            P1F2["P1 (Follower)"]
        end
    end
    P0L -.->|replication| P0F
    P0L -.->|replication| P0F2
    P1L -.->|replication| P1F
    P1L -.->|replication| P1F2

ISR, In-Sync Replicas, is the set of replicas synchronized with the leader. If a follower falls behind, it is removed from the ISR. With acks=all, writes are acknowledged only after all ISR replicas have recorded the message.

min.insync.replicas sets the minimum ISR count. With a replication factor of 3 and min ISR of 2, writes succeed even if one broker fails. If two brokers fail, writes are rejected to protect data consistency.

Leader and Follower Partitions

The replicas of a replicated partition split into two roles. The Leader Partition handles all reads and writes for that partition — both producer writes and consumer reads go to the leader. Follower Partitions only replicate the leader’s log and do not serve client requests. If the leader fails, one of the followers is promoted to leader and takes over the role.

If leaders pile up on a single broker, only that broker bears the load. So Kafka distributes partition leaders evenly across brokers. When a partition is created, the controller places its replicas across brokers in rotation and designates the first broker in each partition’s replica list as the Preferred Leader. Under normal conditions, these preferred leaders are kept as the actual leaders, spreading the leader role across all brokers.

When a broker fails, leadership of the leader partitions it held moves to in-sync followers on other brokers, which can temporarily concentrate leaders on one broker. Once the failed broker recovers, Kafka restores leadership to the preferred leaders (auto.leader.rebalance) to rebalance.

High Watermark

A consumer cannot read every message written to the leader immediately. The leader’s most recent offset is the LEO (Log End Offset), and the point all ISR replicas have caught up to is the High Watermark (HW). Consumers read only up to the HW — to prevent exposing messages not yet safely written to all replicas. A message written with acks=all becomes visible to consumers only after the HW advances to that point.

ZooKeeper and Its Limitations

Before Kafka 3.3, ZooKeeper managed cluster metadata: broker lists, topic/partition configurations, controller election, and ACL information.

The ZooKeeper-based architecture had several problems.

Operational overhead of a separate system. A ZooKeeper cluster (typically 3-5 nodes) must be operated alongside the Kafka cluster. Monitoring, upgrades, and incident response targets double.

Metadata propagation bottleneck. Brokers fetch metadata from ZooKeeper, so as partition counts grow, metadata synchronization takes longer. This slows controller failover recovery in large clusters.

Dual consensus problem. ZooKeeper runs a consensus algorithm (ZAB), while Kafka separately operates ISR-based replication. The two systems can temporarily fall out of sync.

KRaft Mode

KRaft, Kafka Raft, removes ZooKeeper and lets Kafka manage metadata internally. Production use became available in Kafka 3.3, and ZooKeeper mode was removed starting from 4.0.

In KRaft, some nodes take on the Controller role. In production, dedicating separate nodes to the controller is recommended, while in development or small-scale setups a broker can also double as a controller. Controller nodes use the Raft consensus algorithm to agree on a metadata log. Metadata is stored in an internal Kafka topic, eliminating the need for a separate system.

Key changes from ZooKeeper mode:

  • No ZooKeeper cluster. The operational target reduces to Kafka alone.
  • Metadata is managed as an event log. Brokers subscribe to the metadata log and maintain their state. Propagation is faster than polling from ZooKeeper.
  • Controller failover speeds up. The Raft protocol elects a new leader who takes over the metadata log.

Summary

Kafka’s core consists of topics, partitions, and consumer groups. Partitions provide parallelism and ordering guarantees. Consumer groups enable horizontal scaling. Broker replication ensures fault tolerance.

KRaft mode removed ZooKeeper as an external dependency from this structure. Kafka now handles metadata consensus and management on its own.