schema-evolution

name	schema-evolution
description	Evolve database and event schemas without breaking producers or consumers. Use when changing columns, fields, or message contracts in shared systems.

Schema Evolution

Schemas are contracts. Evolve them with expand-contract migrations and registry-enforced compatibility so producers and consumers can deploy independently.

Stack Baseline (2026)

Concern	Recommended
Relational migrations	Postgres 17, Liquibase 4.27, Flyway 10, sqitch
Online DDL	pg_repack, pgroll, gh-ost, Vitess
Event schemas	Avro, Protobuf 3, JSON Schema 2020-12
Registry	Confluent Schema Registry, Apicurio, AWS Glue Registry
Contract tests	Pact 4, Schemathesis, Buf breaking
Streaming	Kafka 3.7 with subject-name compatibility rules
Lakehouse	Iceberg schema evolution, Delta column mapping

When to Use

• Adding, removing, renaming, or retyping a column or event field.
• Splitting or merging entities across services.
• Promoting a v1 contract to v2 with overlapping consumers.
• Backfilling historical data after a model change.

Prerequisites

• Schema registry with compatibility mode set per subject.
• CI gate that runs buf breaking or avro-tools compat.
• Feature flags to gate dual writes/reads.
• Owners list per topic and table.

Instructions

1. Choose the compatibility mode

Mode	Producer can	Consumer can	Use for
Backward	add optional, remove	upgrade first	most consumer-heavy topics
Forward	remove optional, add required	upgrade later	producer-heavy ingestion
Full	only additive optional	either order	shared contracts
None	anything	break	only for v-bumped topics

2. Apply expand-contract for relational changes

sequenceDiagram
    participant App
    participant DB
    Note over DB: Phase 1 - Expand
    App->>DB: ADD COLUMN email_new (nullable)
    App->>DB: dual write old + new
    Note over DB: Phase 2 - Backfill
    App->>DB: UPDATE in batches with throttling
    Note over DB: Phase 3 - Migrate reads
    App->>DB: read from new column
    Note over DB: Phase 4 - Contract
    App->>DB: DROP old column after grace window

3. Write a safe Postgres migration

-- Expand
ALTER TABLE customer ADD COLUMN email_ci citext;
CREATE INDEX CONCURRENTLY ix_customer_email_ci ON customer (email_ci);

-- Backfill (chunked, off-peak)
UPDATE customer SET email_ci = lower(email)
WHERE email_ci IS NULL AND id BETWEEN $1 AND $2;

-- Contract (after consumers migrated + grace window)
ALTER TABLE customer DROP COLUMN email;

Never combine expand + contract in one release.

4. Evolve an Avro event with backward compatibility

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.acme.orders.v1",
  "fields": [
    { "name": "orderId",    "type": "string" },
    { "name": "customerId", "type": "string" },
    { "name": "totalCents", "type": "long" },
    { "name": "currency",   "type": "string", "default": "USD" },
    { "name": "channel",    "type": ["null","string"], "default": null }
  ]
}

New fields require a default. Removing a field requires removing it from all consumers first.

5. Version on breaking change

Bump the namespace (v1 -> v2), publish to a new subject, keep both for the deprecation window, and announce sunset via AsyncAPI 3.0 deprecated: true.

6. Gate in CI

- run: buf breaking --against '.git#branch=main'
- run: schema-registry-cli test-compatibility --subject orders-value --schema schema.avsc

7. Observe after rollout

Track schema-id distribution in the consumer for one week. Alert if >0.5% deserialization errors or if old schema-id traffic exceeds expected tail.

Common Pitfalls

Pitfall	Why it hurts	Mitigation
Renaming columns/fields	Breaks every consumer silently	Add new, dual write, deprecate old
Tightening a type (long -> int)	Forward-incompatible	Keep type, validate at app layer
Required field with no default	Breaks Avro/Protobuf compat	Always provide default or make optional
ALTER TABLE without CONCURRENTLY	Long lock, outage	Use pgroll, gh-ost, or IF NOT EXISTS + concurrent index
Dropping fields too early	Old consumers crash	Enforce grace window via registry metrics
One PR doing expand + contract	Unrecoverable on rollback	Split across releases

Output Format

A migration ticket containing: compatibility mode, schema diff, expand/contract phases with dates, registry IDs, consumer owners notified, rollback SQL, and observability dashboard link.

Authoritative References

• Confluent Schema Registry compatibility - https://docs.confluent.io/platform/current/schema-registry/avro.html
• Buf breaking change detection - https://buf.build/docs/breaking/overview
• pgroll, zero-downtime Postgres migrations - https://github.com/xataio/pgroll
• Iceberg schema evolution - https://iceberg.apache.org/docs/latest/evolution/
• AsyncAPI 3.0 spec - https://www.asyncapi.com/docs/reference/specification/v3.0.0