schema-evolution-compatibility

name	schema-evolution-compatibility
description	Evolves shared data contracts (events, API payloads, DB columns, protobuf/avro) without breaking live consumers — additive-only changes with optional+default fields, NEVER remove/rename/repurpose a field or reuse a protobuf tag / avro position (reserve them with `reserved`/aliases instead), backward vs forward vs full compatibility chosen per producer/consumer upgrade order, expand-then-contract (dual-write/dual-read) migrations for renames and type changes, and a schema registry (Confluent/Buf) wired into CI to mechanically reject incompatible diffs before merge. Tolerant reader, unknown-field preservation, and explicit versioning when a true break is unavoidable.
when_to_use	Changing a schema that something else already reads or writes — adding/removing/renaming a field on a Kafka event, API JSON payload, protobuf/avro/JSON-Schema, or a DB column other services depend on; deciding if a change is safe to deploy and in what order; or wiring registry compat checks into CI. Distinct from design-protobuf-grpc-service (designs the IDL/RPCs from scratch; this evolves an existing one safely) and db-migration-safety (runs the ALTER without locking/downtime; this decides whether the column change breaks readers at all).

When to Use

Reach for this skill when a contract that another process already produces or consumes is changing and you must not break it mid-deploy:

• "Add a field to this Kafka event / API response — will old consumers still parse it?"
• "Rename / remove / change the type of a field that other services read"
• "Which compatibility mode (backward/forward/full) for this Avro subject?"
• "We reused a protobuf field number and a consumer is reading garbage"
• "Deploy producers or consumers first? what's the safe order?"
• "Wire buf breaking / Confluent compat checks into CI so bad diffs get blocked"
• "Migrate a column/field rename with zero downtime across services"

NOT this skill:

• Designing the proto/gRPC service, message shapes, and RPCs from scratch → design-protobuf-grpc-service (this skill evolves an IDL that already has live readers)
• Running the ALTER TABLE itself without locks/downtime (lock-free index, batched backfill, NOT VALID constraints) → db-migration-safety (it makes the DDL safe; this skill decides if the column change breaks consumers)
• Designing the relational schema / normalization / keys → design-relational-schema
• The REST/GraphQL field-type and nullability contract for one endpoint → rest-graphql-contract
• API versioning policy, deprecation headers, pagination contracts → api-design-review / design-api-pagination
• Validating one payload against a schema at the edge (request validation) → build-form-validation / validate-data-quality
• Verifying producer and consumer agree via recorded pacts → contract-testing (it tests the agreement; this skill governs how the schema may change)
• Big phased rewrite/cutover of a whole system → plan-strangler-migration

Steps

1. Pick the compatibility mode from your upgrade order — it's the whole game. Compatibility is asymmetric and defined by who reads data written under the other schema:

   Mode	Guarantees	Allowed change	Upgrade FIRST
   BACKWARD	new consumer reads data from old + new producers	add optional field (w/ default), delete optional field	consumers
   FORWARD	old consumer reads data from new producer	add optional field, delete field that had a default	producers
   FULL	both directions	only add/remove optional fields with defaults	either
   *_TRANSITIVE	same, but vs all prior versions not just the last	—	—

   Default to BACKWARD for events/topics (Confluent's default — consumers lag and replay history, so the new reader must handle old records). Use FORWARD when producers ship ahead of consumers. Use FULL_TRANSITIVE for long-lived event logs you replay from the beginning. The rule of thumb: add a field → forward-safe; remove a field → backward-safe; do both safely → only optional+default.

2. Additive-only is the safe default. Every new field is optional with a default — never required. A new required field breaks every old producer (forward) and every old record (backward) instantly. Concretely:

   • JSON / JSON-Schema: add the key, do NOT add it to required, give consumers a default. Keep additionalProperties permissive (or unevaluatedProperties in 2020-12) so old readers tolerate fields they don't know.
   • protobuf (proto3): every field is already optional; new scalar fields default to 0/""/false. Just append with a fresh field number. Use optional (proto3 explicit presence) when you must distinguish "unset" from "zero".
   • Avro: a new field must carry a "default", or it's neither backward- nor forward-compatible — {"name":"x","type":["null","string"],"default":null}. This is the #1 Avro footgun.

3. NEVER remove, rename, or repurpose a field in place — and NEVER reuse a tag/number/position. Renaming = remove + add to every consumer; changing a field's meaning while keeping its name/number is the worst break because it passes schema checks but silently corrupts data. Reuse of an identifier makes old payloads decode into the wrong field. Reserve instead:

   • protobuf — when you drop field 7 (name email), reserve both so the number and name can never be re-added:

     message User {
       reserved 7, 9 to 11;          // numbers
       reserved "email", "legacy_id"; // names
       string username = 3;
     }
     

   • Avro — never reuse a removed field's name; to rename keep the old name reachable via "aliases": ["old_name"] so readers using the old schema still resolve it.
   • JSON — treat a removed key as permanently retired; never recycle a key name for a different type/meaning.

   A type change (e.g. int32 → string, string → enum) is not additive even if the name stays — it's a remove-and-add. Wire-compatible widenings exist in proto (int32/int64/uint32/bool are interchangeable on the wire; sint*/fixed* are not) but treat them as breaking unless you've verified the exact pair.

4. For a true rename or type change, run expand → migrate → contract (dual-write/dual-read). You cannot atomically change a field across N independently-deployed services. Phase it:

   Phase	Producer	Consumer	DB column
   1 Expand	write BOTH old + new	still reads old	add new col, backfill, dual-write trigger
   2 Migrate	writes both	switch reads to new (fallback to old)	—
   3 Contract	stop writing old; reserve it	reads new only	drop old col (after grace + replay window)

   Each phase is independently deployable and rollback-safe. The grace window between expand and contract must exceed your longest consumer lag + replay/retention window (e.g. Kafka topic retention) so no in-flight or replayed record still needs the old field. The DB column drop is where db-migration-safety takes over.

5. Deploy in the order the compatibility mode dictates — getting this backwards is the classic outage.

   • BACKWARD change (added/removed optional): deploy consumers first, then producers. New consumers can read both shapes; once all consumers handle the new shape, flip producers.
   • FORWARD change: deploy producers first — old consumers tolerate the new field (they ignore unknowns), then upgrade consumers to use it.
   • FULL: either order, but still roll out gradually and watch dead-letter/parse-error metrics during the canary.
   • Never deploy producer and consumer in lockstep assuming atomicity — there is always a window where mixed versions run.

6. Run a schema registry with mechanical compatibility checks, and gate CI on them. Humans miss breaks; the registry doesn't.

   • Confluent Schema Registry (Avro/Protobuf/JSON-Schema over Kafka): set per-subject mode and test the candidate before publishing — curl -X PUT .../config/<subject> -d '{"compatibility":"BACKWARD_TRANSITIVE"}', then POST .../compatibility/subjects/<subject>/versions/latest returns {"is_compatible": true|false}. The Maven/Gradle schema-registry:test-compatibility goal does this in CI.
   • protobuf → buf breaking --against '.git#branch=main' in CI; rules FIELD_NO_DELETE (forces reserved), FIELD_SAME_TYPE, RESERVED_* catch exactly the breaks above. Pair with buf lint.
   • Avro standalone → java -jar avro-tools or the avro-compatibility checker; gate the PR.
   • JSON-Schema → json-schema-diff / oasdiff (for OpenAPI) flag breaking changes.

   Make the check fail the build, not warn. The registry's compatibility setting per subject is the contract; CI is the enforcement.

7. Write consumers as tolerant readers — ignore unknown fields, never hard-fail on them. Forward compatibility depends on the reader's behavior as much as the schema:

   • JSON: don't use a strict/closed deserializer that throws on unknown keys. Jackson → @JsonIgnoreProperties(ignoreUnknown = true) / FAIL_ON_UNKNOWN_PROPERTIES=false; Go encoding/json ignores unknowns by default (avoid DisallowUnknownFields); Pydantic → model_config = ConfigDict(extra="ignore") (NOT "forbid").
   • Preserve, don't drop, unknown fields on a read-modify-write path, or a round-trip through an old service silently deletes data a newer one added. protobuf keeps unknown fields by default; for JSON, capture them (@JsonAnySetter, additionalProperties map) and re-emit. This is the subtle one — a "harmless" old service in the middle of a pipeline strips new fields.
   • Always provide a default when a field is absent; don't assume presence.

8. When a break is genuinely unavoidable, version explicitly — don't mutate in place. Some changes (splitting one field into two, restructuring nesting, semantic redefinition) can't be made compatible. Then:

   • Events: new schema = new subject / new topic (orders.v2) or an explicit schema_version field; run v1 and v2 in parallel; migrate consumers; retire v1 after the replay window. Never silently change v1's meaning.
   • APIs: new path/header version (/v2, Accept: application/vnd.api.v2+json); deprecate v1 with a sunset header and timeline. Versioning is the escape hatch, not the default — additive evolution avoids a version bump for the 90% case.

Common Errors

• Adding a required field. Breaks every old producer and every historical record at once. Fix: optional + default, always.
• Avro field with no default. Silently fails both backward and forward compat. Fix: every Avro field added/removed needs an explicit "default".
• Reusing a protobuf field number (or Avro position). Old payloads decode into the wrong field — type-confused garbage that passes schema checks. Fix: reserved the number AND the name; only ever append fresh numbers.
• Renaming a field in place. It's a delete + add to every consumer simultaneously. Fix: expand→migrate→contract, or Avro aliases.
• Repurposing a field's meaning while keeping its name. Passes all mechanical checks, silently corrupts semantics. Fix: new field; reserve the old one.
• Wrong deploy order for the compat mode. Backward change with producers-first (or forward with consumers-first) → mixed-version outage. Fix: consumers-first for backward, producers-first for forward.
• Strict deserializer that throws on unknown fields. Kills forward compatibility the moment a producer adds a field. Fix: tolerant reader (ignoreUnknown, extra="ignore", no DisallowUnknownFields).
• Dropping unknown fields on read-modify-write. An older service in the pipeline silently erases data newer services added. Fix: preserve and re-emit unknown fields.
• Treating a type widening as free. int32→string or string→enum is a break even with the same name; not all proto widenings are wire-safe. Fix: verify the exact pair or run expand→contract.
• No registry / CI gate. Relying on review to catch breaks. Fix: buf breaking / Confluent compat check that fails the build.
• Checking only against the latest version, not all. A change compatible with v3 but not v1 breaks replay. Fix: *_TRANSITIVE mode for replayable logs.
• Contracting before the replay/retention window passes. Dropping the old field while replayable records still reference it. Fix: grace window > longest consumer lag + topic retention.

Verify

1. Mechanical compat check passes in CI: buf breaking / Confluent is_compatible:true / Avro checker runs on the PR diff and fails the build on an incompatible change — proven by intentionally introducing a remove/rename and watching CI go red.
2. Old-schema read of new data, and vice versa: serialize a record with the new schema, deserialize with the old (forward); serialize with old, read with new (backward) — both succeed, defaults fill absent fields. This is the literal compatibility definition; test it, don't assume it.
3. No required field added, every new field has a default: grep the diff — new fields are optional and defaulted ("default" in Avro, not in JSON required, appended proto numbers).
4. Removed fields are reserved: any dropped proto field has its number AND name in reserved; any renamed Avro field has aliases; no identifier is reused.
5. Tolerant reader confirmed: feed a consumer a payload with an extra unknown field → it parses and ignores it (no exception); on read-modify-write, the unknown field survives the round-trip.
6. Deploy order documented and rehearsed: the rollout plan states consumers-first (backward) or producers-first (forward), and a mixed-version canary shows zero parse errors / dead-letters during the window.
7. Rename via expand→contract, not in place: the migration is staged (dual-write, switch reads, then drop + reserve) and each phase is independently rollback-safe; the old field is dropped only after the replay window.
8. Transitive check for replayable logs: for an event log replayed from offset 0, compat mode is *_TRANSITIVE and a candidate is checked against all prior versions, not just latest.

Done = the change is additive (optional + defaulted) or staged through expand→migrate→contract, no field/tag/position is ever removed-without-reserving or repurposed, the compatibility mode matches the deploy order, consumers are tolerant readers that preserve unknowns, and a schema-registry compat check fails CI on any incompatible diff — all proven by the old↔new round-trip and the red-CI test in checks 1–2.