Command Palette

Search for a command to run...

Hectal
Phase 9Intermediate10 of 18 in Apache Kafka

Kafka Connect, CDC & the Outbox

Moving data in and out of Kafka without custom code: Connect workers, connectors and tasks, source and sink connectors in production, change data capture with Debezium, and the transactional outbox that ends dual writes.

Most organisations don't write every integration by hand. Kafka Connect runs reusable connectors for databases, object storage, search engines and warehouses, and change data capture turns a database's transaction log into a stream of events.

The same machinery solves one of the most common bugs in event-driven systems: writing to the database and to Kafka in two steps that can disagree.

0/4 · 0%
4 topics ~31 min 9 code blocks & diagrams
Start with the first topic
1
9.1

Kafka Connect: Workers, Connectors, and Tasks

Kafka Connect is a framework for streaming data between Kafka and other systems using configuration instead of code. Source connectors import (databases, files, APIs), sink connectors export (search, S3, warehouses, databases). Workers run connectors split into tasks, and Connect stores configs, offsets and status in Kafka topics.

7 min 1 diagram 1 code practice

2
9.2

Running Connect in Production

Run Connect in distributed mode with its internal topics replicated, pin connector versions, size tasks to partitions, configure error tolerance and connector DLQs, monitor task states and lag, and plan for rebalances when connectors or workers change.

7 min 1 code practice

3
9.3

Change Data Capture with Debezium

CDC reads a database's transaction log (PostgreSQL WAL via logical replication, MySQL binlog) and emits every insert, update and delete as an event, in commit order per row, starting with an initial snapshot. It's the reliable way to stream database changes to Kafka without changing application code.

9 min 3 code practice

4
9.4

The Transactional Outbox: Ending Dual Writes

Writing to the database and publishing to Kafka as two separate steps can leave them inconsistent. The outbox pattern writes the business change and an event row in one database transaction; a relay (polling or CDC) publishes outbox rows to Kafka. Events are published if and only if the transaction committed.

8 min 1 diagram 2 code practice