Why Schemas Matter, and Choosing a Format

In one line

Schemas define event structure so producers can't silently break consumers. JSON is readable but unenforced and verbose; Avro is compact with strong evolution rules; Protobuf is compact, fast and popular across languages; JSON Schema adds validation to JSON. Pick based on ecosystem, size, speed and evolution needs.

0/4 · 0%

Think of it like this

A standard shipping container. Every port can handle it because its dimensions are agreed. If one sender ships an odd-shaped crate, cranes everywhere jam. A schema is that agreed shape for data.

Key ideas

01
Without schemas: a producer renames amount to total or changes it from number to string; consumers fail to parse, or worse, parse and silently compute wrong values. Old records in retention still have the old shape, so replays break too.
02
JSON (schemaless): human-readable, universal, but field names repeat in every record (bigger, slower), types are loose (numbers vs strings), and nothing enforces the contract.
03
Avro: binary, compact (no field names in records), schema required to read (resolved via schema ID), rich evolution rules with defaults; strong in the Kafka/Hadoop ecosystem.
04
Protobuf: binary, compact and fast, field numbers identify fields so renames are safe, excellent multi-language codegen; optional and reserved fields help evolution. JSON Schema: keeps JSON on the wire but validates structure; evolution rules are looser and harder to reason about.
05
Typical sizes for the same event: JSON ~1.0×, JSON Schema ~1.0× (same wire format), Avro ~0.3–0.5×, Protobuf ~0.3–0.5×; binary formats also serialize faster.

Code & diagrams

formats.txttext

                 JSON        JSON Schema   Avro             Protobuf
Wire size        large       large         small            small
Speed            slow        slow          fast             fastest
Readable         yes         yes           no (tooling)     no (tooling)
Enforced schema  no          yes           yes              yes
Evolution        ad hoc      loose rules   defaults, rich   field numbers, rich
Codegen          optional    optional      yes              yes (best multi-language)
Registry         optional    yes           yes              yes

Explain it without notes

Why do schemas matter more for Kafka than for a REST API?

Practice

Encode one of your events in JSON, Avro and Protobuf and compare sizes.

Trade-offs

↔
Binary formats save space and enforce contracts at the cost of tooling to read records and a registry dependency.

Done when you can

I can compare JSON, Avro, Protobuf and JSON Schema and choose one for a use case.