The Log on Disk: Segments and Indexes

In one line

Each partition is a directory of segment files; only the newest (active) segment is written. Each segment has a .log file of record batches plus sparse .index (offset → file position) and .timeindex (timestamp → offset) files, so Kafka can find any offset or time quickly and delete old data a whole segment at a time.

0/6 · 0%

Think of it like this

A diary kept in numbered notebooks. You only write in the current notebook; when it's full you start a new one. Each notebook's first page lists roughly where each week starts, so you can jump to any date without reading everything, and old notebooks are thrown away whole.

Key ideas

01
Directory orders-3/ contains 00000000000000000000.log, .index, .timeindex, then 00000000000000521377.log and so on; the file name is the segment's base offset.
02
Rolling: a new segment starts when the active one reaches segment.bytes (1 GB default) or segment.ms (7 days) elapses. Retention and compaction act only on closed segments, never on the active one.
03
Sparse offset index: an entry every index.interval.bytes (4 KB) of log data. To find offset N, binary-search the segment files by base offset, binary-search the index for the nearest earlier entry, then scan forward a few KB. Indexes are memory-mapped.
04
Time index: maps timestamps to offsets, used for time-based retention and offsetsForTimes() (seek to a point in time, for example --reset-offsets --to-datetime).
05
Offsets are per partition: offset 42 in partition 0 and offset 42 in partition 1 are unrelated records. A record is identified by (topic, partition, offset).

Code & diagrams

segments.shbash

ls -la /var/lib/kafka/data/commerce.orders-3/
00000000000000000000.index      10485760
00000000000000000000.log       1073741211
00000000000000000000.timeindex  10485756
00000000000000521377.index      10485760
00000000000000521377.log         42191870   <- active segment
00000000000000521377.timeindex  10485756
leader-epoch-checkpoint
partition.metadata

kafka-dump-log.sh --files 00000000000000521377.log --print-data-log | head -4
baseOffset: 521377 lastOffset: 521390 count: 14 baseSequence: 88 lastSequence: 101 producerId: 4001
  producerEpoch: 0 partitionLeaderEpoch: 7 isTransactional: false position: 0
  CreateTime: 1727520002113 size: 2211 magic: 2 compresscodec: zstd crc: 3319201

Explain it without notes

How does Kafka find offset 700,000 in a partition quickly?

Practice

Use kafka-dump-log.sh on a segment in your lab and identify the batch headers: producer ID, sequence numbers, compression codec.

Trade-offs

↔
Smaller segments make retention more precise but create more files and index overhead.

Done when you can

I can describe segments, the active segment, sparse offset and time indexes.