data-retention

name	data-retention
description	Design data retention policies covering deletion schedules, archival tiers, compliance requirements, and right-to-erasure workflows. Outputs retention schedules, automated deletion jobs, anonymization strategies, and audit trails.
argument-hint	["data types","compliance requirements (GDPR/CCPA/HIPAA)","storage costs","query requirements for old data"]
allowed-tools	Read, Write, Bash

Data Retention

Data retention balances legal compliance, storage cost, and query requirements. Too little retention violates business needs; too much creates compliance risk and storage cost. Every piece of data needs an explicit retention policy.

Process

1. Inventory all data — what data exists, where it's stored, who owns it.
2. Classify by sensitivity — PII, financial, operational, logs, analytics.
3. Determine legal requirements — GDPR (3 years financial, erasure on request), HIPAA (6 years), CCPA.
4. Define retention periods — by data type, not one-size-fits-all.
5. Design archival tiers — hot (active DB) → warm (compressed archive) → cold (S3 Glacier).
6. Automate deletion — manual processes are forgotten; schedule deletion jobs.
7. Implement right-to-erasure — GDPR Article 17 requires deletion within 30 days of request.
8. Audit the audit trail — deletion logs must themselves be retained.

Output Format

Retention Policy Definition

# data-retention-policy.yaml
version: "1.0"
last_reviewed: "2024-01-15"
next_review: "2025-01-15"

data_categories:
  user_accounts:
    description: "User registration data, preferences, authentication"
    contains_pii: true
    retention_period: "active + 3 years after account deletion"
    legal_basis: "GDPR Art. 6(1)(b) - contract; 7-year financial retention"
    storage: [postgres, s3]
    deletion_method: anonymize  # Replace with anonymous values
    right_to_erasure: true
    notes: "Retain transaction records 7 years for tax; anonymize PII"
  
  order_records:
    description: "Purchase history, invoices, shipping details"
    contains_pii: true  # Shipping address, name
    retention_period: "7 years from transaction date"
    legal_basis: "Tax law requires 7-year financial record retention"
    storage: [postgres, s3_archive]
    deletion_method: anonymize_pii_keep_financial
    right_to_erasure: false  # Overridden by financial law
    notes: "Shipping address anonymized; order amounts retained for tax"
  
  
     
       
       
     
     [, ]
     
     
        
  
  
     
       
     
     
     [, ]
     
     
     
  
  
     
     
     
     
     [, ]
     

Automated Deletion Jobs

# retention/deletion_jobs.py
import logging
from datetime import datetime, timezone, timedelta
import boto3
import psycopg2
from dataclasses import dataclass
from typing import Optional

logger = logging.getLogger(__name__)

@dataclass
class DeletionResult:
    job_name: str
    rows_affected: int
    execution_time_ms: float
    errors: list[str]
    dry_run: bool

class RetentionJobRunner:
    def __init__(self, db_url: str, dry_run: bool = True):
        self.db_url = db_url
        self.dry_run = dry_run
    
    def run_all(self) -> list[DeletionResult]:
        """Run all scheduled retention jobs."""
        jobs = [
            self.delete_old_event_logs,
            self.anonymize_deleted_users,
            self.archive_old_orders,
            self.purge_expired_sessions,
            self.pseudonymize_old_access_logs,
        ]
        
        results = []
        for job in jobs:
            try:
                result = job()
                results.append(result)
                logger.info()
             Exception  e:
                logger.error()
                results.append(DeletionResult(
                    job_name=job.__name__,
                    rows_affected=,
                    execution_time_ms=,
                    errors=[(e)],
                    dry_run=.dry_run,
                ))
        
         results
    
     () -> DeletionResult:
        
        cutoff = datetime.now(timezone.utc) - timedelta(days=)  
        
        start = datetime.now()
        rows = 
        
        conn = psycopg2.connect(.db_url)
        :
             conn.cursor()  cur:
                 .dry_run:
                    cur.execute(
                        ,
                        (cutoff,)
                    )
                    rows = cur.fetchone()[]
                    logger.info()
                :
                    
                     :
                        cur.execute(, (cutoff,))
                        batch = cur.rowcount
                        conn.commit()
                        rows += batch
                         batch == :
                            
            
            ._log_deletion(conn, , rows, cutoff)
        :
            conn.close()
        
        elapsed = (datetime.now() - start).total_seconds() * 
         DeletionResult(, rows, elapsed, [], .dry_run)
    
     () -> DeletionResult:
        
        cutoff = datetime.now(timezone.utc) - timedelta(days=)
        
        conn = psycopg2.connect(.db_url)
        rows = 
        :
             conn.cursor()  cur:
                 .dry_run:
                    cur.execute(
                        ,
                        (cutoff,)
                    )
                    rows = cur.fetchone()[]
                :
                    cur.execute(, (cutoff,))
                    rows = cur.rowcount
                    conn.commit()
        :
            conn.close()
        
        elapsed = 
         DeletionResult(, rows, elapsed, [], .dry_run)
    
     ():
        
         .dry_run:
            
        
         conn.cursor()  cur:
            cur.execute(, (table, rows, cutoff))
            conn.commit()



 ():
    s3 = boto3.client()
    
    s3.put_bucket_lifecycle_configuration(
        Bucket=bucket,
        LifecycleConfiguration={
            : [
                {
                    : ,
                    : ,
                    : {: },
                    : [
                        {
                            : ,
                            :    
                        },
                        {
                            : ,
                            :     
                        },
                    ],
                    : {: },             
                },
                {
                    : ,
                    : ,
                    : {: },
                    : [
                        {
                            : ,
                            :         
                        }
                    ],
                    
                },
                {
                    : ,
                    : ,
                    : {: },
                    : {: },
                },
            ]
        }
    )

Right-to-Erasure Workflow

# gdpr/erasure_workflow.py
class ErasureRequestProcessor:
    """
    Process GDPR Article 17 right-to-erasure requests.
    Must complete within 30 days; best practice is 5-7 business days.
    """
    
    def __init__(self, db, s3_client, email_service):
        self.db = db
        self.s3 = s3_client
        self.email = email_service
    
    async def process_erasure_request(self, user_id: str, request_id: str) -> dict:
        results = {}
        
        # 1. Verify identity and log request
        await self.db.execute(
            "INSERT INTO erasure_requests (request_id, user_id, requested_at, status) VALUES ($1, $2, NOW(), 'processing')",
            request_id, user_id
        )
        
        # 2. Anonymize user account (keep shell for referential integrity)
        await self.db.execute("""
            UPDATE users SET
                email = 'erased_' || $1 || '@gdpr.invalid',
                name = 'Erased User',
                phone = NULL,
                date_of_birth = NULL,
                profile_photo_url = NULL,
                is_erased = true,
                erased_at = NOW()
            WHERE id = $1
        """, user_id)
        results["user_account"] = "anonymized"
        
        # 3. Delete behavioral data (not subject to financial retention)
        deleted_events = await self.db.fetchval(
            "WITH d AS (DELETE FROM event_logs WHERE user_id = $1 RETURNING 1) SELECT COUNT(*) FROM d",
            user_id
        )
        results["event_logs"] = f"deleted  rows"
        
        
         .db.execute(, user_id)
        results[] = 
        
        
         ._delete_s3_data(user_id)
        results[] = 
        
        
         .db.execute(, user_id)
         .db.execute(, user_id)
        results[] = 
        
        
         .db.execute(
            ,
            request_id, json.dumps(results)
        )
        
        
        
         .email.send_erasure_confirmation(request_id=request_id)
        
         results
    
      ():
        
        paginator = .s3.get_paginator()
        
          page  paginator.paginate(
            Bucket=settings.DATA_LAKE_BUCKET,
            Prefix=
        ):
             obj  page.get(, []):
                .s3.delete_object(Bucket=settings.DATA_LAKE_BUCKET, Key=obj[])

Rules

• Every data element needs a retention period — "keep forever" is a policy decision, not an oversight.
• Financial data overrides erasure — GDPR right-to-erasure yields to legal financial record-keeping obligations.
• Anonymize instead of delete where referential integrity matters — preserve IDs, replace PII with anonymous values.
• Deletion jobs must be idempotent — running twice should produce the same result as running once.
• Audit the deletions — deletion audit logs must themselves be retained (typically 7 years).
• Test erasure workflows — regularly verify that right-to-erasure requests actually remove data from all systems.
• Staged deletion — soft-delete → grace period → anonymize → hard delete. Never skip stages.
• Cold storage is not deletion — archiving to Glacier is not erasure; still subject to right-to-erasure.
• Don't forget derived data — analytics aggregates computed from PII may themselves contain PII.
• Document legal basis for every retention period — when a regulator asks, you need the citation.