Schema Evolution and Compatibility

In one line

Compatibility modes decide which changes are allowed. BACKWARD: new consumers can read old data (add optional fields with defaults, delete fields). FORWARD: old consumers can read new data. FULL: both. Transitive variants check against all versions. Renames and type changes are breaking; handle them with new fields or a new topic.

0/4 · 0%

Think of it like this

Updating a paper form. Adding an optional box is fine; old forms still work and old clerks just ignore the new box. Removing a box that clerks rely on, or changing "age" from a number to words, breaks things.

Key ideas

01
BACKWARD (Confluent default): consumers using the new schema can read data written with the previous schema. Allowed: add fields with defaults, remove fields. Upgrade consumers first, then producers.
02
FORWARD: consumers using the old schema can read data written with the new schema. Allowed: add fields (old readers ignore them), remove fields that had defaults. Upgrade producers first.
03
FULL: both directions; only add or remove fields with defaults. Safest for many independent consumers. _TRANSITIVE variants check against all previous versions, not just the latest, which matters for long retention and replays.
04
Breaking changes: renaming a field (in Avro; Protobuf uses field numbers so renames are wire-safe), changing a type incompatibly (int → string), changing meaning or units silently. Approach: add a new field, dual-write both for a while, migrate consumers, then remove the old one; or publish to a new versioned topic.
05
Semantic changes aren't caught by any registry: changing amount from cents to rupees with the same type is compatible by schema and wrong in reality. Document meaning and units in the schema.

Code & diagrams

user-v1-v2.avscjson

// v1
{ "type": "record", "name": "User", "fields": [
    { "name": "id",   "type": "long" },
    { "name": "name", "type": "string" }
]}
// v2: adds an optional field with a default -> BACKWARD, FORWARD and FULL compatible
{ "type": "record", "name": "User", "fields": [
    { "name": "id",    "type": "long" },
    { "name": "name",  "type": "string" },
    { "name": "email", "type": ["null", "string"], "default": null }
]}

compat-check.shbash

curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @user-v3-email-required.json \
  http://registry:8081/compatibility/subjects/users-value/versions/latest
{"is_compatible":false,
 "messages":["READER_FIELD_MISSING_DEFAULT_VALUE: email"]}

Interview problem

The problem

Adding email to the user event

Version 1 of a user event has id and name. Version 2 adds email. Can old consumers keep working? Then discuss incompatible changes: making email required, renaming name to fullName, and changing id from long to string.

Explain it without notes

Explain BACKWARD vs FORWARD compatibility and the upgrade order each implies.

Practice

Set a subject to FULL_TRANSITIVE and try registering: add optional field, remove a field without default, change a type.

Trade-offs

↔
Stricter compatibility limits change freedom but protects many unknown consumers and long-retention replays.

Done when you can

I can evolve schemas safely and handle breaking changes with dual-writes or new topics.

Command Palette