tg-migrations

name	tg-migrations
description	Django migration best practices for the World of Darkness application. Use when creating data migrations, handling schema changes with existing data, squashing old migrations, testing migrations in CI/CD, or planning multi-step migrations for breaking changes. Triggers on migration work, database schema changes, or data transformation tasks.

Migration Best Practices

Core Principles

1. Never edit applied migrations - Once in production, migrations are immutable
2. Test before deployment - Test both forward and reverse migrations
3. Keep migrations focused - One logical change per migration
4. Document complex migrations - Explain non-obvious data transformations
5. Plan multi-step migrations - For breaking changes, maintain backward compatibility

Data Migrations for Schema Changes

When to Use Data Migrations

Create data migrations when you need to:

• Transform existing data during schema changes
• Populate new fields with calculated values
• Migrate data between models (e.g., JSONField to related models)
• Clean up or normalize existing data

Three-Step Pattern for New Required Fields

Step 1: Add field as nullable

operations = [
    migrations.AddField(
        model_name='character',
        name='generation',
        field=models.IntegerField(null=True, blank=True),
    ),
]

Step 2: Populate with data

def populate_generation(apps, schema_editor):
    Character = apps.get_model('characters', 'Character')
    for char in Character.objects.filter(character_type='vampire', generation__isnull=True):
        char.generation = 13
        char.save(update_fields=['generation'])

def reverse_populate(apps, schema_editor):
    Character = apps.get_model('characters', 'Character')
    Character.objects.update(generation=None)

operations = [
    migrations.RunPython(populate_generation, reverse_populate),
]

Step 3: Make non-nullable

operations = [
    migrations.AlterField(
        model_name='character',
        name='generation',
        field=models.IntegerField(default=13),
    ),
]

Data Migration Best Practices

Use Historical Models

def migrate_data(apps, schema_editor):
    # CORRECT - uses historical model state
    Character = apps.get_model('characters', 'Character')

    # WRONG - uses current model which may have different fields
    from characters.models import Character  # Don't do this!

Handle Large Datasets

def migrate_large_dataset(apps, schema_editor):
    Character = apps.get_model('characters', 'Character')

    # Use iterator() to prevent loading all into memory
    for char in Character.objects.iterator(chunk_size=1000):
        char.new_field = calculate_value(char)
        char.save(update_fields=['new_field'])

    # Or use bulk operations
    updates = []
    for char in Character.objects.iterator(chunk_size=1000):
        char.new_field = calculate_value(char)
        updates.append(char)
        if len(updates) >= 1000:
            Character.objects.bulk_update(updates, ['new_field'])
            updates = []
    if updates:
        Character.objects.bulk_update(updates, ['new_field'])

Always Provide Reverse Migrations

operations = [
    migrations.RunPython(
        forward_migration,
        reverse_migration,  # Always provide this
    ),
]

Make Migrations Idempotent

def migrate_data(apps, schema_editor):
    """Migration that can safely run multiple times."""
    Model = apps.get_model('app', 'Model')
    
    # Check before creating
    if not Model.objects.filter(special_flag=True).exists():
        # Perform migration
        pass

Migration Squashing

When to Squash

• App has 50+ migrations
• Migrations are all applied to production
• Old migrations contain outdated data transformations

Squashing Process

# 1. Identify range
python manage.py showmigrations characters

# 2. Create squashed migration
python manage.py squashmigrations characters 0001 0050

# 3. Test on fresh database
python manage.py migrate characters

# 4. Verify and run tests
python manage.py check
python manage.py test characters

After Squashing

After squashed migration is in production for several weeks:

# Remove old migration files
rm characters/migrations/0001_initial.py
# ... through 0050

# Remove replaces attribute from squashed migration
# Edit and remove: replaces = [...]

Common Migration Patterns

Renaming a Field (Safe)

# Step 1: Add new field
operations = [migrations.AddField('Model', 'new_name', field=...)]

# Step 2: Copy data
operations = [
    migrations.RunPython(
        lambda apps, se: apps.get_model('app', 'Model').objects.update(new_name=F('old_name'))
    ),
]

# Step 3: Remove old field
operations = [migrations.RemoveField('Model', 'old_name')]

Changing Field Type

# Step 1: Add new field with new type
operations = [migrations.AddField('Model', 'field_new', new_field_type)]

# Step 2: Transform and copy data
def transform_data(apps, schema_editor):
    Model = apps.get_model('app', 'Model')
    for obj in Model.objects.iterator():
        obj.field_new = transform(obj.field_old)
        obj.save(update_fields=['field_new'])

# Step 3: Remove old field
operations = [migrations.RemoveField('Model', 'field_old')]

# Step 4: Rename new field
operations = [migrations.RenameField('Model', 'field_new', 'field_old')]

Testing Migrations

CI/CD Migration Tests

# Check for conflicts
python manage.py makemigrations --check --dry-run

# Test forward
python manage.py migrate

# Test backward (last 3)
python manage.py migrate characters 0000
python manage.py migrate characters

Migration Test Class

from django.test import TransactionTestCase
from django.core.management import call_command

class MigrationTestCase(TransactionTestCase):
    migrate_from = None
    migrate_to = None
    app_name = 'characters'

    def setUp(self):
        if self.migrate_from:
            call_command('migrate', self.app_name, self.migrate_from)
        super().setUp()

    def migrate(self, target):
        call_command('migrate', self.app_name, target)

    def test_migration_forward_backward(self):
        if not self.migrate_to or not self.migrate_from:
            self.skipTest("Migration range not specified")
        self.migrate(self.migrate_to)
        self.migrate(self.migrate_from)

Pre-Deployment Checklist

Before Creating Migration

• Check if change requires multi-step migration
• Plan data migration if schema affects existing data
• Consider performance impact on large tables

Before Committing

• Migration name is descriptive
• Complex logic has comments
• Reverse migration provided for RunPython
• Tested on development database

Before Deploying

• Tested on production database copy
• Verified performance on realistic dataset
• Documented in deployment guide if manual steps needed

Troubleshooting

Migration Conflicts

# Django auto-detects and creates merge migration
python manage.py makemigrations

Fake Migrations (Emergency)

# Mark as applied without running
python manage.py migrate app_name 0042 --fake

# Only use when database state matches migration

Rolling Back

# Rollback to previous
python manage.py migrate app_name 0041_previous

# Fix bad migration, create new one
python manage.py makemigrations
python manage.py migrate