backward-compatibility-rules

name	Backward Compatibility Rules
description	Comprehensive guide to backward compatibility rules for APIs, databases, and data contracts with migration strategies and testing approaches

Backward Compatibility Rules

What is Backward Compatibility?

Definition: New version works with old clients without requiring changes.

Example

API v1: GET /users → { id, name }
API v2: GET /users → { id, name, email }  (added field)

Old client (expects v1) calls v2 API:
→ Receives { id, name, email }
→ Ignores 'email' field
→ Works! ✅ Backward compatible

Evolution Design

Version 1.0 → 1.1 → 1.2 → 2.0
         ↓      ↓      ↓      ↓
    Compatible Compatible Breaking

---

Why Backward Compatibility Matters

1. Avoid Breaking Existing Integrations

Without Backward Compatibility:

Deploy new API version
→ All clients break immediately
→ Emergency rollback
→ Coordinate with all clients to update
→ Big bang migration (risky!)

With Backward Compatibility:

Deploy new API version
→ Old clients continue working
→ Clients migrate gradually
→ No coordination needed

2. Gradual Migration (Not Big Bang)

Timeline:

Month 0: Deploy v2 (backward compatible)
Month 1: 20% of clients migrated
Month 3: 50% of clients migrated
Month 6: 80% of clients migrated
Month 12: 100% migrated, deprecate v1

3. Reduce Coordination Overhead

Without:

• Coordinate deployment with all clients
• Schedule maintenance window
• Rollback plan if issues

With:

• Deploy anytime
• Clients migrate when ready
• No coordination needed

4. Customer Trust (Stable APIs)

Trust:

• "This API won't break my integration"
• "I can upgrade when I'm ready"
• "No surprises"

---

Compatibility Levels

Backward Compatible (New Server, Old Client)

Definition: Old client works with new server

Example:

Old client expects: { id, name }
New server returns: { id, name, email }

Client ignores 'email' → Works ✅

Forward Compatible (Old Server, New Client)

Definition: New client works with old server

Example:

New client expects: { id, name, email? }
Old server returns: { id, name }

Client handles missing 'email' → Works ✅

Full Compatible (Both Directions)

Definition: Works both ways

Example:

v1: { id, name }
v2: { id, name, email? }  (optional)

Old client + new server: Ignores 'email' ✅
New client + old server: Handles missing 'email' ✅

Breaking Change (Incompatible)

Definition: Doesn't work

Example:

Old client expects: { id, name }
New server returns: { id, firstName, lastName }

Client looks for 'name' → Not found → Breaks ❌

---

REST API Backward Compatibility

Safe Changes

Add New Endpoint

Before:
GET /users

After:
GET /users
GET /projects  (new)

Compatible: ✅ Old clients don't call new endpoint

Add Optional Field to Request

Before:
POST /users
{ "name": "John" }

After:
POST /users
{ "name": "John", "email": "[e-mail]" }  (email optional)

Compatible: ✅ Old clients don't send 'email', server handles missing field

Add Field to Response

Before:
GET /users → { "id": "123", "name": "John" }

After:
GET /users → { "id": "123", "name": "John", "email": "[e-mail]" }

Compatible: ✅ Old clients ignore 'email'

Add Enum Value (at End)

Before:
status: "ACTIVE" | "INACTIVE"

After:
status: "ACTIVE" | "INACTIVE" | "PENDING"

Compatible: ✅ Old clients may not handle "PENDING", but schema is valid

Relax Validation (Accept More)

Before:
name: minLength 5

After:
name: minLength 1  (less strict)

Compatible: ✅ Old clients sending 5+ chars still work

Breaking Changes

Remove Endpoint

Before:
GET /users

After:
(endpoint removed)

Breaking: ❌ Old clients calling endpoint get 404

Remove Field from Response

Before:
GET /users → { "id": "123", "name": "John" }

After:
GET /users → { "id": "123" }  ('name' removed)

Breaking: ❌ Old clients expect 'name' field

Add Required Field to Request

Before:
POST /users
{ "name": "John" }

After:
POST /users
{ "name": "John", "email": "[e-mail]" }  (email required)

Breaking: ❌ Old clients don't send 'email', get 400 error

Change Field Type

Before:
age: integer

After:
age: string

Breaking: ❌ Old clients send integer, server expects string

Rename Field

Before:
{ "name": "John" }

After:
{ "fullName": "John" }

Breaking: ❌ Old clients look for 'name', not found

Change Semantics

Before:
amount: 100  (dollars)

After:
amount: 10000  (cents, changed meaning!)

Breaking: ❌ Same field, different meaning

---

Database Schema Compatibility

Safe Migrations

Add Nullable Column

-- Safe: Existing rows get NULL
ALTER TABLE users ADD COLUMN email VARCHAR(255);

Add Table

-- Safe: Doesn't affect existing tables
CREATE TABLE projects (
  id UUID PRIMARY KEY,
  name VARCHAR(255)
);

Add Index

-- Safe: Improves performance, doesn't change data
CREATE INDEX idx_users_email ON users(email);

Breaking Migrations

Drop Column (Requires Multi-Step Migration)

-- Breaking: Existing queries fail
ALTER TABLE users DROP COLUMN name;

-- Multi-step migration:
-- Step 1: Stop writing to column
-- Step 2: Deploy code that doesn't read column
-- Step 3: Drop column

Rename Column (Use Views/Aliases)

-- Breaking: Existing queries fail
ALTER TABLE users RENAME COLUMN name TO full_name;

-- Multi-step migration:
-- Step 1: Add new column
ALTER TABLE users ADD COLUMN full_name VARCHAR(255);

-- Step 2: Copy data
UPDATE users SET full_name = name;

-- Step 3: Create view with old name
CREATE VIEW users_v1 AS
SELECT id, full_name AS name FROM users;

-- Step 4: Migrate clients to new column
-- Step 5: Drop view and old column

Change Column Type

-- Breaking: Data may not fit
ALTER TABLE users ALTER COLUMN age TYPE VARCHAR(10);

-- Multi-step migration:
-- Step 1: Add new column
ALTER TABLE users ADD COLUMN age_str VARCHAR(10);

-- Step 2: Copy and convert data
UPDATE users SET age_str = age::VARCHAR;

-- Step 3: Migrate clients
-- Step 4: Drop old column, rename new column

---

Event/Message Compatibility

Safe Changes

Add Optional Field

Before:
{ "id": "123", "name": "John" }

After:
{ "id": "123", "name": "John", "email": "[e-mail]" }

Compatible: ✅ Old consumers ignore 'email'

Add New Event Type

Before:
user.created

After:
user.created
user.deleted  (new)

Compatible: ✅ Old consumers don't subscribe to new event

Breaking Changes

Remove Field

Before:
{ "id": "123", "name": "John" }

After:
{ "id": "123" }  ('name' removed)

Breaking: ❌ Old consumers expect 'name'

Change Field Type

Before:
{ "age": 30 }  (integer)

After:
{ "age": "30" }  (string)

Breaking: ❌ Old consumers expect integer

Rename Field

Before:
{ "name": "John" }

After:
{ "fullName": "John" }

Breaking: ❌ Old consumers look for 'name'

---

GraphQL Compatibility

Safe Changes

Add New Field to Type

Before:
type User {
  id: ID!
  name: String!
}

After:
type User {
  id: ID!
  name: String!
  email: String  # New field
}

Compatible: ✅ Old queries don't request 'email'

Add New Type

After:
type Project {
  id: ID!
  name: String!
}

Compatible: ✅ Old queries don't use new type

Add New Query/Mutation

After:
type Query {
  users: [User!]!
  projects: [Project!]!  # New query
}

Compatible: ✅ Old clients don't call new query

Breaking Changes

Remove Field

Before:
type User {
  id: ID!
  name: String!
}

After:
type User {
  id: ID!
  # 'name' removed
}

Breaking: ❌ Old queries requesting 'name' fail

Change Field Type

Before:
type User {
  age: Int!
}

After:
type User {
  age: String!  # Changed type
}

Breaking: ❌ Old queries expect Int

Change Field Arguments

Before:
users(limit: Int): [User!]!

After:
users(limit: Int!, offset: Int!): [User!]!  # 'offset' required

Breaking: ❌ Old queries don't provide 'offset'

---

Deprecation Strategy

Mark as Deprecated

In Docs:

## GET /users/:id/profile

**Deprecated:** Use GET /users/:id instead.
**Sunset Date:** 2024-12-31

In Code:

/**
 * @deprecated Use getUser() instead. Will be removed in v2.0.0.
 */
function getUserProfile(id) {
  // ...
}

In Headers:

HTTP/1.1 200 OK
Deprecation: true
Sunset: Sat, 31 Dec 2024 23:59:59 GMT
Link: <https://docs.example.com/migration>; rel="deprecation"

Provide Migration Guide

Example:

# Migration Guide: /users/:id/profile → /users/:id

## What's Changing
The `/users/:id/profile` endpoint is deprecated. Use `/users/:id` instead.

## Why
Consolidating user endpoints for simplicity.

## How to Migrate

### Before
```javascript
const profile = await fetch('/api/users/123/profile');

After

const user = await fetch('/api/users/123');

Timeline

• 2024-06-01: Deprecation announced
• 2024-12-31: Endpoint will stop working

### Set Sunset Date (6-12 Months)

**Timeline:**

Month 0: Announce deprecation Month 3: Warn users (emails, headers) Month 6: Final warning Month 12: Sunset (stop working)

### Warn Clients (in Response Headers)

HTTP/1.1 200 OK Deprecation: true Sunset: Sat, 31 Dec 2024 23:59:59 GMT Warning: 299 - "This endpoint is deprecated. Use /v2/users instead. See https://docs.example.com/migration"

### Monitor Usage

**Track Deprecated Endpoint Usage:**
```javascript
app.get('/users/:id/profile', (req, res) => {
  // Log usage
  metrics.increment('deprecated.users_profile.calls');
  
  // Return response with deprecation headers
  res.set('Deprecation', 'true');
  res.set('Sunset', 'Sat, 31 Dec 2024 23:59:59 GMT');
  res.json(profile);
});

Dashboard:

Deprecated Endpoint Usage:
/users/:id/profile
- Calls/day: 1,000 (down from 5,000 last month)
- Top clients: mobile-app (500), web-app (300), partner-api (200)
- Sunset date: 2024-12-31 (6 months remaining)

Remove After Sunset

Process:

1. Sunset date reached
2. Return 410 Gone
3. Monitor for errors
4. After 1 month, remove code

410 Gone Response:

HTTP/1.1 410 Gone
Content-Type: application/json

{
  "error": "This endpoint has been removed. Use /v2/users instead.",
  "migrationGuide": "https://docs.example.com/migration"
}

---

Multi-Step Migrations

Example: Rename Field

Step 1: Add New Field, Both Fields Populated

// API returns both fields
{
  "name": "John Doe",      // Old field (deprecated)
  "fullName": "John Doe"   // New field
}

Step 2: Clients Migrate to New Field

Month 1-3: Clients update to use 'fullName'

Step 3: Stop Populating Old Field (Still Present)

// API returns both fields, but old field is null
{
  "name": null,            // Old field (deprecated, not populated)
  "fullName": "John Doe"   // New field
}

Step 4: Remove Old Field

// API returns only new field
{
  "fullName": "John Doe"   // New field
}

---

API Versioning Strategies

URL Versioning (/v1/, /v2/)

Format:

/api/v1/users
/api/v2/users

Pros:

• Clear and visible
• Easy to route
• Can run multiple versions

Cons:

• URL changes
• Maintain multiple codebases

Header Versioning (Accept-Version: v1)

Format:

GET /api/users
Accept-Version: v1

Pros:

• URL doesn't change
• More RESTful

Cons:

• Less visible
• Harder to test

No Versioning (Only Non-Breaking Changes)

Strategy:

• Never make breaking changes
• Always backward compatible

Pros:

• Simple
• No migration

Cons:

• Technical debt accumulates

---

Testing Backward Compatibility

Contract Tests (Pact)

See Contract Testing skill for details

Integration Tests with Old Clients

Example:

describe('Backward compatibility', () => {
  test('v1 client works with v2 API', async () => {
    // v1 client expects { id, name }
    const response = await v1Client.get('/users/123');
    
    expect(response).toHaveProperty('id');
    expect(response).toHaveProperty('name');
    // v2 API may return additional fields (email), but v1 client ignores them
  });
});

Schema Compatibility Checks (Automated)

OpenAPI:

oasdiff breaking old.yaml new.yaml

JSON Schema:

const isCompatible = checkBackwardCompatibility(oldSchema, newSchema);
if (!isCompatible) {
  throw new Error('Breaking change detected');
}

Canary Deployments

Process:

1. Deploy new version to 5% of traffic
2. Monitor errors
3. If no errors, increase to 25%
4. Continue until 100%

---

Compatibility Checking Tools

OpenAPI: oasdiff, openapi-diff

See OpenAPI Governance skill

GraphQL: graphql-inspector

Install:

npm install -g @graphql-inspector/cli

Check:

graphql-inspector diff old.graphql new.graphql

Output:

✖ Field 'name' was removed from object type 'User' (BREAKING)
✔ Field 'email' was added to object type 'User'

Protobuf: buf breaking

Install:

brew install bufbuild/buf/buf

Check:

buf breaking --against .git#branch=main

JSON Schema: schema validators

Example:

function checkBackwardCompatibility(oldSchema, newSchema) {
  // Check if all required fields in old schema are still required
  for (const field of oldSchema.required) {
    if (!newSchema.required.includes(field)) {
      return false;  // Breaking: required field removed
    }
  }
  
  // Check if field types changed
  for (const [field, oldType] of Object.entries(oldSchema.properties)) {
    const newType = newSchema.properties[field];
    if (newType && newType.type !== oldType.type) {
      return false;  // Breaking: type changed
    }
  }
  
  return true;  // Compatible
}

---

Communicating Breaking Changes

Changelog (What Changed)

Example:

# Changelog

## v2.0.0 (2024-06-01) - BREAKING CHANGES

### Breaking Changes
- **Removed:** `GET /users/:id/profile` endpoint. Use `GET /users/:id` instead.
- **Changed:** `POST /users` now requires `email` field.
- **Renamed:** Response field `name` renamed to `fullName`.

### Migration Guide
See https://docs.example.com/migration/v1-to-v2

## v1.2.0 (2024-05-01)

### Added
- New field `avatar` in user response (optional, backward compatible)

Migration Guide (How to Update)

See "Provide Migration Guide" section above

Deprecation Notice (When Will It Break)

See "Deprecation Strategy" section above

Direct Outreach (Email Major Clients)

Email Template:

Subject: Action Required: API v1 Deprecation

Hi [Client],

We're writing to inform you that API v1 will be deprecated on December 31, 2024.

What's changing:
- Endpoint /users/:id/profile will be removed
- Use /users/:id instead

Action required:
- Update your integration by December 31, 2024
- See migration guide: https://docs.example.com/migration

Need help?
- Reply to this email
- Schedule a call: https://calendly.com/api-team

Thank you,
API Team

---

Handling Inevitable Breaking Changes

Major Version Bump (v1 → v2)

Process:

1. Create v2 API (breaking changes)
2. Maintain v1 API (for transition)
3. Announce deprecation of v1
4. Give 6-12 months notice
5. Sunset v1

Maintain v1 for Transition Period

Timeline:

Month 0: Launch v2
Month 0-12: Maintain both v1 and v2
Month 12: Sunset v1

Clear Migration Path

Provide:

• Migration guide (step-by-step)
• Code examples (before/after)
• Tools to help migrate
• Support (email, calls)

Tools to Help Migrate

Example:

# Migration tool
./migrate-to-v2.sh

# Scans code for v1 API calls
# Suggests v2 equivalents
# Optionally auto-updates code

---

Real-World Examples

Stripe API Evolution

Versioning:

• Header-based: Stripe-Version: 2024-01-15
• Backward compatible within version
• Breaking changes = new version

Deprecation:

• 6-12 months notice
• Detailed migration guides
• Direct outreach to major customers

AWS API Backward Compatibility

Policy:

• Never break existing APIs
• Add new APIs for new features
• Deprecate old APIs (but keep working)

GraphQL Schema Evolution

Best Practices:

• Add fields (don't remove)
• Deprecate fields (don't remove immediately)
• Use @deprecated directive

---

Best Practices Checklist

- [ ] All changes reviewed for compatibility
- [ ] Automated compatibility checks in CI
- [ ] Deprecation process defined (6-12 months)
- [ ] Migration guides for breaking changes
- [ ] Versioning strategy documented
- [ ] Monitor deprecated endpoint usage
- [ ] Direct outreach to major clients
- [ ] Test with old clients
- [ ] Canary deployments for risky changes

---

Implementation

Compatibility Check Scripts

See "Compatibility Checking Tools" section

CI/CD Integration

GitHub Actions:

- name: Check backward compatibility
  run: |
    # OpenAPI
    oasdiff breaking origin/main:openapi.yaml HEAD:openapi.yaml
    
    # GraphQL
    graphql-inspector diff origin/main:schema.graphql HEAD:schema.graphql
    
    # Fail if breaking changes
    if [ $? -ne 0 ]; then
      echo "Breaking changes detected"
      exit 1
    fi

Deprecation Header Middleware

Express:

function deprecationMiddleware(sunsetDate, migrationUrl) {
  return (req, res, next) => {
    res.set('Deprecation', 'true');
    res.set('Sunset', sunsetDate);
    res.set('Link', `<${migrationUrl}>; rel="deprecation"`);
    res.set('Warning', `299 - "This endpoint is deprecated. See ${migrationUrl}"`);
    
    // Log usage
    metrics.increment('deprecated.endpoint.calls', {
      path: req.path,
      client: req.headers['user-agent']
    });
    
    next();
  };
}

// Usage
app.get('/users/:id/profile',
  deprecationMiddleware('Sat, 31 Dec 2024 23:59:59 GMT', 'https://docs.example.com/migration'),
  (req, res) => {
    // Handle request
  }
);

---

Summary

Quick Reference

Backward Compatibility: New version works with old clients

Why:

• Avoid breaking integrations
• Gradual migration
• Reduce coordination
• Customer trust

Safe Changes (REST API):

• Add endpoint
• Add optional field to request
• Add field to response
• Add enum value
• Relax validation

Breaking Changes:

• Remove endpoint/field
• Add required field to request
• Change field type
• Rename field
• Change semantics

Database:

• Safe: Add nullable column, add table, add index
• Breaking: Drop column, rename column, change type

Deprecation:

• Mark as deprecated
• Migration guide
• 6-12 months notice
• Monitor usage
• Remove after sunset

Multi-Step Migration:

1. Add new field, populate both
2. Clients migrate
3. Stop populating old field
4. Remove old field

Versioning:

• URL: /v1/, /v2/
• Header: Accept-Version: v1
• No versioning (only non-breaking)

Testing:

• Contract tests (Pact)
• Integration tests with old clients
• Schema compatibility checks
• Canary deployments

Tools:

• OpenAPI: oasdiff
• GraphQL: graphql-inspector
• Protobuf: buf breaking
• JSON Schema: custom validators

Communicating:

• Changelog
• Migration guide
• Deprecation notice
• Direct outreach

Best Practices

Version Control

• Use semantic versioning (SemVer)

• Tag all releases with version numbers

[ ] Document version compatibility matrix

• Maintain backward compatibility

[ ] Use @deprecated directive for breaking changes

• [ ]

Communication

• Communicate changes early

• Provide migration guides

• Hold breaking change meetings

[ ] Create data contracts with consumers

[ ] Document breaking changes clearly

[ ]

Testing

• Test backward compatibility

• Test with production-like data

[ ] Test migration scripts thoroughly

• Monitor test coverage

[ ]

Migration

• Use zero-downtime migration pattern

• Use multi-step migrations for breaking changes

[ ] Backfill data before removing old columns

• Test migrations in staging environment

[ ]

Documentation

[ ] Document all breaking changes with dates

[ ] Document breaking changes clearly

[ ] Provide migration guides for consumers

[ ] Document version compatibility

[ ]

Monitoring

[ ] Monitor breaking change impacts

[ ] Track consumer adoption of new schema

[ ] Set up dashboards for schema health

[ ] [ ]

Prevention

[ ] Use data contracts for all shared data

[ ] Enforce schema validation at source

[ ] [ ] Implement CI/CD schema checks

[ ]

Checklist

[ ] Use semantic versioning

[ ] Implement zero-downtime migrations

[ ] Backfill data before removing old columns

[ ] Test backward compatibility

[ ] Have rollback procedures ready

[ ] Monitor schema drift metrics

[ ] Track migration success rates

[ ] Document all breaking changes

[ ] Set up change notifications

[ ] Test with production-like data

[ ] Monitor test coverage

[ ] Optimize schema validation overhead

[ ] Cache schema definitions

[ ] Use efficient validation libraries

[ ] Monitor schema performance impact

[ ] Establish schema ownership

[ ] Create schema review process

[ ] Define schema lifecycle

[ ] Plan schema deprecation strategy

[ ] Set up incident response for violations

[ ] Test schema validation logic

[ ] Train team on backward compatibility

[ ]