schema-evolution-playbook

name	schema-evolution-playbook
description	Step-by-step playbook for evolving Avro/Protobuf/JSON Schema event schemas without breaking consumers, covering compatibility modes, migration patterns, and registry operations.

Schema Evolution Playbook

When to Use This

A producer needs to change an event schema — add a field, rename one, remove one, change a type. Use this playbook before any schema change lands in production to ensure zero consumer breakage.

Compatibility Mode Selection

Change type	Backward compatible	Forward compatible	Full compatible
Add optional field with default	Yes	No	No
Remove optional field	No	Yes	No
Add + remove optional fields	No	No	Yes (both modes)
Rename field	Never compatible — use aliases instead
Change field type	Almost never — add new field, deprecate old

Default recommendation: set the registry to BACKWARD — consumers can upgrade independently after the producer ships.

Step-by-Step Migration

1. Identify all consumers — query the schema registry's subject references or run kafka-consumer-groups.sh --list to enumerate active groups for the topic.
2. Classify the change — use the table above to pick the minimum compatibility mode needed.
3. Register the candidate schema — use the registry's POST /compatibility endpoint to check before any topic write:

   curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
     --data '{"schema": "<escaped-json>"}' \
     https://registry/compatibility/subjects/order-events-value/versions/latest
   

4. Add default values for new fields — Avro requires a default for backward-compatible additions; null is acceptable for optional fields but explicit application defaults carry intent.
5. For breaking changes — dual-write transition:
   • Introduce a new topic (order-events-v2) or a new subject strategy (e.g. TopicRecordNameStrategy).
   • Run both producers simultaneously until all consumers migrate.
   • Tombstone old topic (set retention to short window) only after all consumer-group offsets have passed the last old-schema message.
6. Deprecate old fields in the schema — use doc strings or a custom "deprecated": true property; do not delete for at least two release cycles.
7. Update the schema registry soft-delete / hard-delete policy — only hard-delete after all consumers confirm upgrade.

Renaming Fields Without Breaking

Use Avro aliases:

{
  "name": "userId",
  "type": "string",
  "aliases": ["user_id"],
  "doc": "Renamed from user_id in v3. Alias preserved for old consumers."
}

Protobuf: keep the field number; change the name only in the .proto source — wire format is number-keyed.

Pitfalls

• Registering a schema without checking compatibility first — the registry will reject it at write time, not before, causing producer failures mid-stream.
• Removing a field that a consumer reads without a default — consumers throw deserialization errors on historical messages replayed from offset 0.
• Using TopicNameStrategy with multiple record types in one topic — evolving one breaks the other; prefer TopicRecordNameStrategy or separate topics.
• Skipping the compatibility check step because "it's just adding a field" — null defaults are easy to forget, and missing them causes backward-incompatibility silently.
• Deleting a schema version before all consumers have processed every message serialized under it — even with short retention, reprocessing from a snapshot can surface old versions.

See Also

• ../../agents/kafka-pipeline-engineer.md — schema registry + compatibility/evolution ownership
• ../../agents/streaming-architect.md — delivery semantics and topology decisions