Command Palette

Search for a command to run...

Hectal
Phase 8Intermediate9 of 18 in Apache Kafka

Schemas & Data Contracts

Why schemas matter, JSON vs Avro vs Protobuf vs JSON Schema, Schema Registry subjects and IDs, backward/forward/full compatibility, safe evolution, and topic-per-entity vs topic-per-event strategies.

A Kafka topic is an API between teams that never talk at deploy time. Without schemas, one producer's "small change" breaks consumers in production, possibly days later when they replay history.

This phase makes events contracts: typed, versioned, compatibility-checked at build time, and evolved without breaking anyone.

0/4 · 0%
4 topics ~29 min 6 code blocks & diagrams
Start with the first topic
1
8.1

Why Schemas Matter, and Choosing a Format

Schemas define event structure so producers can't silently break consumers. JSON is readable but unenforced and verbose; Avro is compact with strong evolution rules; Protobuf is compact, fast and popular across languages; JSON Schema adds validation to JSON. Pick based on ecosystem, size, speed and evolution needs.

7 min 1 code practice

2
8.2

Schema Registry: Subjects, IDs, and Serializers

A schema registry stores versioned schemas under subjects (usually <topic>-value), assigns each schema a global ID, and enforces compatibility on registration. Serializers put the schema ID in each record (a magic byte plus 4-byte ID), and deserializers fetch and cache the schema to decode.

7 min 2 code practice

3
8.3

Schema Evolution and Compatibility

Compatibility modes decide which changes are allowed. BACKWARD: new consumers can read old data (add optional fields with defaults, delete fields). FORWARD: old consumers can read new data. FULL: both. Transitive variants check against all versions. Renames and type changes are breaking; handle them with new fields or a new topic.

8 min 2 code practice

4
8.4

Topic Strategy: One Topic, Many Topics, or Types in the Payload

Organise events as one topic per entity with multiple event types (ordering across types, union schemas), one topic per event type (clean schemas and ACLs, no cross-type order), or one catch-all topic (avoid). Decide by ordering needs, consumer filtering cost, retention and ownership.

7 min 1 code practice