schema-evolution-compatibility

name	schema-evolution-compatibility
description	Evolves shared data contracts (events, API payloads, DB columns, protobuf/avro) without breaking live consumers — additive-only changes with optional+default fields, NEVER remove/rename/repurpose a field or reuse a protobuf tag / avro position (reserve them with `reserved`/aliases instead), backward vs forward vs full compatibility chosen per producer/consumer upgrade order, expand-then-contract (dual-write/dual-read) migrations for renames and type changes, and a schema registry (Confluent/Buf) wired into CI to mechanically reject incompatible diffs before merge. Tolerant reader, unknown-field preservation, and explicit versioning when a true break is unavoidable.
when_to_use	Changing a schema that something else already reads or writes — adding/removing/renaming a field on a Kafka event, API JSON payload, protobuf/avro/JSON-Schema, or a DB column other services depend on; deciding if a change is safe to deploy and in what order; or wiring registry compat checks into CI. Distinct from design-protobuf-grpc-service (designs the IDL/RPCs from scratch; this evolves an existing one safely) and db-migration-safety (runs the ALTER without locking/downtime; this decides whether the column change breaks readers at all).

When to Use

Reach for this skill when a contract that another process already produces or consumes is changing and you must not break it mid-deploy:

• "Add a field to this Kafka event / API response — will old consumers still parse it?"
• "Rename / remove / change the type of a field that other services read"
• "Which compatibility mode (backward/forward/full) for this Avro subject?"
• "We reused a protobuf field number and a consumer is reading garbage"
• "Deploy producers or consumers first? what's the safe order?"
• "Wire buf breaking / Confluent compat checks into CI so bad diffs get blocked"
• "Migrate a column/field rename with zero downtime across services"

NOT this skill:

• Designing the proto/gRPC service, message shapes, and RPCs from scratch → design-protobuf-grpc-service (this skill evolves an IDL that already has live readers)
• Running the ALTER TABLE itself without locks/downtime (lock-free index, batched backfill, NOT VALID constraints) → db-migration-safety (it makes the DDL safe; this skill decides if the column change breaks consumers)
• Designing the relational schema / normalization / keys → design-relational-schema
• The REST/GraphQL field-type and nullability contract for one endpoint → rest-graphql-contract
• API versioning policy, deprecation headers, pagination contracts → api-design-review / design-api-pagination
• Validating one payload against a schema at the edge (request validation) → build-form-validation / validate-data-quality
• Verifying producer and consumer agree via recorded pacts → contract-testing (it tests the agreement; this skill governs how the schema may change)
• Big phased rewrite/cutover of a whole system → plan-strangler-migration

Steps

1. Pick the compatibility mode from your upgrade order — it's the whole game. Compatibility is asymmetric and defined by who reads data written under the other schema:

   Mode	Guarantees	Allowed change	Upgrade FIRST
   BACKWARD	new consumer reads data from old + new producers	add optional field (w/ default), delete optional field	consumers
   FORWARD	old consumer reads data from new producer	add optional field, delete field that had a default	producers
   FULL	both directions	only add/remove optional fields with defaults	either
   *_TRANSITIVE	same, but vs all prior versions not just the last	—	—

   Default to BACKWARD for events/topics (Confluent's default — consumers lag and replay history, so the new reader must handle old records). Use FORWARD when producers ship ahead of consumers. Use FULL_TRANSITIVE for long-lived event logs you replay from the beginning. The rule of thumb: add a field → forward-safe; remove a field → backward-safe; do both safely → only optional+default.

2. Additive-only is the safe default. Every new field is optional with a default — never required. A new required field breaks every old producer (forward) and every old record (backward) instantly. Concretely:

   • JSON / JSON-Schema: add the key, do NOT add it to required, give consumers a default. Keep additionalProperties permissive (or unevaluatedProperties in 2020-12) so old readers tolerate fields they don't know.
   • protobuf (proto3): every field is already optional; new scalar fields default to 0/""/false. Just append with a fresh field number. Use optional (proto3 explicit presence) when you must distinguish "unset" from "zero".

Common Errors

• Adding a required field. Breaks every old producer and every historical record at once. Fix: optional + default, always.
• Avro field with no default. Silently fails both backward and forward compat. Fix: every Avro field added/removed needs an explicit "default".
• Reusing a protobuf field number (or Avro position). Old payloads decode into the wrong field — type-confused garbage that passes schema checks. Fix: reserved the number AND the name; only ever append fresh numbers.
• Renaming a field in place. It's a delete + add to every consumer simultaneously. Fix: expand→migrate→contract, or Avro aliases.
• Repurposing a field's meaning while keeping its name. Passes all mechanical checks, silently corrupts semantics. Fix: new field; reserve the old one.
• Wrong deploy order for the compat mode. Backward change with producers-first (or forward with consumers-first) → mixed-version outage. Fix: consumers-first for backward, producers-first for forward.
• Strict deserializer that throws on unknown fields. Kills forward compatibility the moment a producer adds a field. Fix: tolerant reader (ignoreUnknown, extra="ignore", no DisallowUnknownFields).
• Dropping unknown fields on read-modify-write. An older service in the pipeline silently erases data newer services added. Fix: preserve and re-emit unknown fields.
• Treating a type widening as free. int32→string or string→enum is a break even with the same name; not all proto widenings are wire-safe. Fix: verify the exact pair or run expand→contract.
• No registry / CI gate. Relying on review to catch breaks. Fix: buf breaking / Confluent compat check that fails the build.
• Checking only against the latest version, not all. A change compatible with v3 but not v1 breaks replay. Fix: *_TRANSITIVE mode for replayable logs.
• Contracting before the replay/retention window passes. Dropping the old field while replayable records still reference it. Fix: grace window > longest consumer lag + topic retention.

Verify

1. Mechanical compat check passes in CI: buf breaking / Confluent is_compatible:true / Avro checker runs on the PR diff and fails the build on an incompatible change — proven by intentionally introducing a remove/rename and watching CI go red.
2. Old-schema read of new data, and vice versa: serialize a record with the new schema, deserialize with the old (forward); serialize with old, read with new (backward) — both succeed, defaults fill absent fields. This is the literal compatibility definition; test it, don't assume it.
3. No required field added, every new field has a default: grep the diff — new fields are optional and defaulted ("default" in Avro, not in JSON required, appended proto numbers).
4. Removed fields are reserved: any dropped proto field has its number AND name in reserved; any renamed Avro field has aliases; no identifier is reused.
5. Tolerant reader confirmed: feed a consumer a payload with an extra unknown field → it parses and ignores it (no exception); on read-modify-write, the unknown field survives the round-trip.
6. Deploy order documented and rehearsed: the rollout plan states consumers-first (backward) or producers-first (forward), and a mixed-version canary shows zero parse errors / dead-letters during the window.
7. Rename via expand→contract, not in place: the migration is staged (dual-write, switch reads, then drop + reserve) and each phase is independently rollback-safe; the old field is dropped only after the replay window.
8. Transitive check for replayable logs: for an event log replayed from offset 0, compat mode is *_TRANSITIVE and a candidate is checked against all prior versions, not just latest.

Done = the change is additive (optional + defaulted) or staged through expand→migrate→contract, no field/tag/position is ever removed-without-reserving or repurposed, the compatibility mode matches the deploy order, consumers are tolerant readers that preserve unknowns, and a schema-registry compat check fails CI on any incompatible diff — all proven by the old↔new round-trip and the red-CI test in checks 1–2.