The Apache Kafka Roadmap

What Kafka really is (a distributed, replicated commit log), why teams adopt it over direct REST calls, its core architecture, a local KRaft lab, and how to think about events.

How topics are configured and named, why partitions exist, how keys decide ordering and placement, and how to reason about partition count instead of guessing a number.

Inside the producer: serializers, partitioner, record accumulator and sender thread; batching and compression; acks and durability; retries, timeouts, the idempotent producer and in-flight requests.

The poll loop, offsets and commits, consumer groups and scaling limits, rebalancing protocols (eager, cooperative, static membership, the new KIP-848 protocol), and processing models that don't break ordering.

Leaders, followers and the ISR, high watermarks, leader election and unclean elections, the replication factor + acks + min.insync.replicas combination, and surviving an availability-zone failure.

At-most-once, at-least-once and exactly-once precisely; Kafka transactions and read-process-write; where exactly-once ends (external APIs, databases, emails); and the inbox pattern for idempotent consumers.

Retries with backoff and jitter, retry topics and dead-letter topics, poison messages, consumer lag and backpressure, ordering guarantees end to end, and hot partitions.

Production-grade Spring Kafka producers and consumers: JSON events with keys and headers, idempotence and compression, manual acks, error handlers, DLTs, graceful shutdown, multi-group pipelines, metrics, and testing with Testcontainers.

Why schemas matter, JSON vs Avro vs Protobuf vs JSON Schema, Schema Registry subjects and IDs, backward/forward/full compatibility, safe evolution, and topic-per-entity vs topic-per-event strategies.

Moving data in and out of Kafka without custom code: Connect workers, connectors and tasks, source and sink connectors in production, change data capture with Debezium, and the transactional outbox that ends dual writes.

Log segments and sparse indexes, why Kafka is fast (sequential I/O, page cache, zero-copy, batching), JVM and OS resources, time- and size-based retention, log compaction and tombstones, and tiered storage.

KStream, KTable and GlobalKTable; stateless and stateful operations; RocksDB state stores, changelogs, standby replicas and recovery; windowing; event time and late data; joins, co-partitioning and repartitioning.

What brokers do with each request, why Kafka replaced ZooKeeper with KRaft, how the controller quorum and metadata log work, controller failover, migrating from ZooKeeper, and day-2 cluster operations.

Security (TLS, SASL, ACLs, encryption), quotas and multi-tenancy, capacity planning with real arithmetic, large messages and the claim-check pattern, and disaster recovery across regions.

The broker, producer and consumer metrics that matter, lag observability that tells you whether consumers are catching up, a debugging playbook for client and broker errors, a throughput-incident tree, and failure injection for every scenario.

Event-driven microservices with clear event ownership, sagas with compensation, event sourcing, CQRS read models, and replaying or backfilling history safely.

Kafka vs Redis, RabbitMQ, SQS and Pub/Sub; anti-patterns and when not to use Kafka; FAANG-style designs for analytics pipelines, ride-hailing, activity feeds, payments, log aggregation and CDC platforms; and the e-commerce capstone.

The full Kafka question bank from beginner to staff with model answers, the 40 hands-on builds, study plans by experience level, and the final readiness checklist.