Topics and Topic Configuration

In one line

A topic is a named, partitioned log with its own configuration: partition count, replication factor, retention, cleanup policy, min.insync.replicas, segment size and more. Topic-level settings override broker defaults, and a few of them decide durability and cost more than anything else.

0/5 · 0%

Think of it like this

A set of filing cabinets for one kind of document. You decide how many drawers (partitions), how many copies to keep in other buildings (replication factor), and how long to keep files before shredding them (retention).

Key ideas

01
Create explicitly: kafka-topics.sh --create --topic orders --partitions 12 --replication-factor 3 --config min.insync.replicas=2 --config retention.ms=604800000. Disable auto-creation in production (auto.create.topics.enable=false) so typos don't create topics with default settings.
02
Key configs: retention.ms (default 7 days) and retention.bytes (per partition, default unlimited); cleanup.policy (delete, compact, or both); min.insync.replicas; segment.bytes (1 GB) and segment.ms; max.message.bytes; message.timestamp.type; compression.type (producer keeps what producers send); unclean.leader.election.enable (false by default).
03
Change configs at runtime with kafka-configs.sh --alter --entity-type topics --entity-name orders --add-config retention.ms=259200000. Partition count can only be increased (kafka-topics.sh --alter --partitions 24), and doing so changes which partition existing keys map to.
04
Replication factor: 3 is the production default (tolerates one broker loss with min.insync.replicas=2 while still accepting writes). RF can be changed only by a partition reassignment.
05
Manage topics as code (Terraform provider, Strimzi KafkaTopic resources, or GitOps scripts) with reviews, so configuration is visible and reproducible.

Code & diagrams

topic-configs.shbash

kafka-topics.sh --bootstrap-server $B --create --topic payments \
  --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=1209600000 \
  --config cleanup.policy=delete \
  --config compression.type=zstd

kafka-configs.sh --bootstrap-server $B --describe --entity-type topics --entity-name payments
Dynamic configs for topic payments are:
  compression.type=zstd sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=zstd, DEFAULT_CONFIG:compression.type=producer}
  min.insync.replicas=2 ...
  retention.ms=1209600000 ...

kafka-configs.sh --bootstrap-server $B --alter --entity-type topics --entity-name payments \
  --add-config retention.ms=2592000000
Completed updating config for topic payments.

When it breaks

Auto topic creation enabled in production

What you see

A producer typo creates oders with 1 partition and replication factor 1; events silently flow there, unreplicated and unconsumed.

Fix & prevent

auto.create.topics.enable=false; create topics through reviewed automation; alert on unknown topics.

Explain it without notes

Which topic settings most affect durability and cost?

Practice

Create a topic, then change its retention to 1 hour and watch old segments get deleted (use a small segment.ms in the lab).

Trade-offs

↔
Longer retention enables replay and new consumers but multiplies disk cost by the replication factor.

Done when you can

I can create and alter topics with the key configs and explain each.

Command Palette