Tiered Storage: Long Retention Without Huge Brokers

In one line

Tiered storage (KIP-405, production-ready since Kafka 3.9) keeps recent segments on broker disks and moves older closed segments to object storage like S3. Retention can grow to months or years cheaply, brokers stay small, and adding or replacing brokers no longer means copying terabytes.

0/6 · 0%

Think of it like this

A busy office keeps this month's files in desk drawers and older files in a cheap off-site archive. Staff still retrieve old files when needed, just a bit slower, and new desks don't need to hold years of paperwork.

Key ideas

01
How it works: a remote storage manager plugin (for example an S3 implementation) uploads closed segments; brokers keep local.retention.ms of data locally while retention.ms governs total retention. Reads of old offsets fetch from remote storage transparently.
02
Benefits: storage cost (object storage is far cheaper than replicated block volumes, and replication is handled by the object store), faster broker replacement and rebalancing (only local data moves), and isolation of cold reads from hot page cache.
03
Limitations to know: remote reads have higher latency; compacted topics aren't supported for tiering; you need a remote storage plugin; check your version's release notes for supported features and operational tools.
04
Enable: remote.log.storage.system.enable=true on brokers with a plugin configured, then per topic remote.storage.enable=true, local.retention.ms (hot), retention.ms (total).
05
Managed services (Confluent, MSK, Aiven and others) offer their own tiered storage implementations, and newer Kafka-compatible systems build on object storage entirely.

Code & diagrams

tiered.shbash

# broker (plus the remote storage manager plugin's own settings)
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=<plugin class>
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager

# topic: 1 day hot on brokers, 1 year total
kafka-configs.sh --bootstrap-server $B --alter --entity-type topics --entity-name audit.events \
  --add-config remote.storage.enable=true,local.retention.ms=86400000,retention.ms=31536000000

Explain it without notes

How does tiered storage change broker scaling and recovery?

Practice

Estimate storage cost for 50 MB/s retained 90 days with RF 3 on block storage vs tiered (1 day local).

Trade-offs

↔
Much cheaper long retention and faster operations, at the cost of slower historical reads and another dependency.

Done when you can

I can explain tiered storage, when to use it and its limitations.