Command Palette

Search for a command to run...

Hectal
Phase 10Advanced11 of 18 in Apache Kafka

Storage & Performance Internals

Log segments and sparse indexes, why Kafka is fast (sequential I/O, page cache, zero-copy, batching), JVM and OS resources, time- and size-based retention, log compaction and tombstones, and tiered storage.

Kafka's performance comes from a handful of deliberate design choices: append-only files, letting the operating system cache them, sending bytes from disk to network without copying them through the JVM, and batching everything.

Understanding the storage layout also explains retention, compaction and the newer tiered storage, which changes how much data a cluster can economically keep.

0/6 · 0%
6 topics ~44 min 8 code blocks & diagrams
Start with the first topic
1
10.1

The Log on Disk: Segments and Indexes

Each partition is a directory of segment files; only the newest (active) segment is written. Each segment has a .log file of record batches plus sparse .index (offset → file position) and .timeindex (timestamp → offset) files, so Kafka can find any offset or time quickly and delete old data a whole segment at a time.

7 min 1 code practice

2
10.2

Why Kafka Is Fast: Sequential I/O, Page Cache, Zero-Copy

Kafka appends sequentially, lets the OS page cache hold hot data (not the JVM heap), serves consumers straight from the page cache with sendfile zero-copy, and batches and compresses everything end to end, so one broker can move hundreds of MB/s on modest hardware.

7 min 1 diagram 1 code practice

3
10.3

Broker Resources: JVM, GC, Disks, File Handles, Network

A broker needs a right-sized heap with a low-pause collector, lots of free RAM for page cache, fast disks (several log dirs or JBOD), high file-descriptor and memory-map limits, and network bandwidth sized for produce, replication and consumer fan-out combined.

7 min 1 code practice

4
10.4

Retention: Time, Size, and Cleanup Policies

With cleanup.policy=delete, whole closed segments are deleted once they're older than retention.ms or when the partition exceeds retention.bytes. With compact, Kafka keeps the latest record per key. compact,delete does both. Retention is approximate at segment granularity and drives storage cost.

7 min 1 code practice

5
10.5

Log Compaction and Tombstones

Compacted topics keep at least the latest record for each key, removing older values in the background. A record with a null value (a tombstone) marks a key as deleted and is itself removed after delete.retention.ms. Compaction turns a topic into a replayable, latest-state table: ideal for user profiles, configuration, CDC and Kafka Streams changelogs.

9 min 2 code practice

6
10.6

Tiered Storage: Long Retention Without Huge Brokers

Tiered storage (KIP-405, production-ready since Kafka 3.9) keeps recent segments on broker disks and moves older closed segments to object storage like S3. Retention can grow to months or years cheaply, brokers stay small, and adding or replacing brokers no longer means copying terabytes.

7 min 1 code practice