ml-batch-processing

# ML Batch Processing Pattern ## Classification - **Domain**: Computer Science, AI/ML - **Category**: ML System Design Patterns - **Novelty**: 6/10 (established pattern with modern evolution) - **Practitioner Evidence**: 10/10 (Google, industry standard) ## Mental Model Batch processing decouples prediction from real-time requests by pre-computing predictions on scheduled intervals. Like meal prep for the week instead of cooking each meal on-demand—you process predictions in bulk during off-peak hours, store results, and serve them instantly when requested. ## When to Use - Predictions needed for all users/items at regular intervals (daily recommendations, weekly reports) - Training data arrives in batches rather than continuously - Cost optimization prioritized over real-time freshness (batch = cheaper compute) - Predictions can tolerate staleness (hours/days old acceptable) - High-throughput scenarios where latency isn't critical ## Core Framework ### 1. Schedule Determination **Identify prediction cadence based on business requirements** - Daily batch: Nightly recommendation refresh for morning users - Hourly batch: Stock predictions updating each trading hour - Weekly batch: Monthly subscription churn predictions - Event-triggered: Batch after data warehouse ETL completion ### 2. Data Ingestion Setup **Configure batch data pipeline from sources to ML system** - Extract from data warehouse/data lake (BigQuery, Snowflake, S3) - Apply feature transformations matching training pipeline - Validate schema consistency with model expectations - Handle missing values using same imputation as training ### 3. Distributed Processing Architecture **Parallelize prediction computation across infrastructure** - Use MapReduce/Spark for horizontal scaling across datasets - Partition data by entity (user_id, product_id) for independent processing - Configure batch size based on memory constraints (1K-100K records/batch) - Implement checkpointing for fault tolerance on long-running jobs ### 4. Model Serving Configuration **Deploy model in batch-optimized inference mode** - Load model once per batch job (avoid reload overhead) - Use batch prediction APIs (TensorFlow batch_predict, PyTorch batch inference) - Enable GPU batching for deep learning models (32-512 samples/batch) - Leverage model compilation (TensorRT, ONNX) for throughput optimization ### 5. Prediction Storage Design **Store pre-computed predictions for fast lookup** - Key-value store for individual lookups (Redis, DynamoDB: user_id → prediction) - Columnar storage for analytics (Parquet, BigQuery: all predictions for analysis) - Include metadata (model_version, prediction_timestamp, confidence_score) - Set TTL based on batch frequency (1.5x batch interval for overlap) ### 6. Keyed Predictions Pattern **Enable distributed batch prediction with result matching** - Attach unique keys to input records (primary keys, composite keys) - Preserve keys through prediction pipeline (input → features → predictions) - Join predictions back to original entities using keys - Handle missing predictions (timeouts, errors) with fallback logic ### 7. Monitoring & Alerting **Track batch job health and prediction quality** - Job completion metrics (duration, throughput, failure rate) - Data quality checks (null rate, distribution shifts, schema violations) - Model performance monitoring (prediction distribution, confidence intervals) - Alerting on batch failures or stale predictions (SLA breaches) ## Practical Application ### E-commerce Recommendation System **Problem**: Generate personalized product recommendations for 10M users **Batch Solution**: 1. Nightly job extracts user behavior (purchases, views, clicks) from data warehouse 2. Spark cluster processes 10M users in parallel (10K users/partition × 1K partitions) 3. Recommendation model generates top-100 products per user (batch size: 256 users) 4. Predictions stored in Redis with 36-hour TTL (user_id → [product_ids + scores]) 5. Web app reads pre-computed recommendations in <5ms (vs. 200ms real-time inference) ### Credit Card Fraud Detection (Batch Component) **Problem**: Update fraud risk scores for all accounts daily **Batch Solution**: 1. Daily batch (3am) processes all 50M accounts using last 30 days transactions 2. Feature engineering pipeline computes aggregates (transaction velocity, geography patterns) 3. XGBoost model scores all accounts (1M accounts/minute on 100-node cluster) 4. Risk scores stored in Aurora DB (account_id, risk_score, score_date) 5. Real-time transactions query batch scores + apply real-time rules for final decision ## Edge Cases & Nuances **Cold Start Problem**: New users/items without predictions - Fallback to popularity-based or demographic-based defaults - Trigger on-demand prediction for high-value new entities - Include new entities in next batch cycle with minimal features **Prediction Staleness**: Batch predictions lag reality - Hybrid approach: batch for stable predictions + real-time updates for high-velocity features - Monitor staleness impact on business metrics (click-through rate decay over time) - Decrease batch interval if staleness hurts performance (daily → hourly) **Batch Job Failures**: Incomplete or failed batch runs - Implement idempotent batch jobs (can safely re-run without duplicates) - Use transactional writes to prediction store (all-or-nothing semantics) - Maintain previous batch predictions as fallback until new batch succeeds **Cost vs. Freshness Tradeoff**: More frequent batches = higher cost - Profile actual prediction change rate (how often do top-10 recommendations shift?) - A/B test batch frequencies to measure impact on engagement metrics - Use event-triggered batches for critical updates (product catalog changes) ## Anti-Patterns **Batch for Latency-Critical Applications**: Using batch for fraud detection that must block transactions in real-time **Over-Engineering Batch Infrastructure**: Building distributed system for 10K records processable on single machine **Ignoring Data Freshness Requirements**: Daily batches for inventory predictions when stock changes hourly **No Fallback Strategy**: System breaks when batch job fails with no stale predictions ## Trade-offs **Batch vs. Online Inference**: - Batch: Lower cost (bulk processing), higher latency (stale predictions), simpler ops (scheduled jobs) - Online: Higher cost (per-request compute), lower latency (fresh predictions), complex ops (SLA-driven) **Batch Frequency**: - More frequent (hourly): Fresher predictions, higher compute cost, more operational complexity - Less frequent (daily): Stale predictions, lower cost, simpler ops, higher storage requirements **Distributed vs. Single-Node**: - Distributed: Scales to billions of records, complex infrastructure, slower for small datasets - Single-node: Simple, fast for <10M records, memory/compute constraints, no fault tolerance ## Related Frameworks - **Streaming ML Pattern**: Continuous prediction updates from streaming data (complements batch) - **Online Learning Pattern**: Incremental model updates as new data arrives (batch retraining alternative) - **Lambda Architecture**: Batch layer + speed layer for hybrid batch/streaming systems - **Feature Store Pattern**: Centralized feature computation for batch and online consistency ## Practitioner Sources - **Google ML Design Patterns** (Lakshmanan et al.): Batch Serving pattern (#17), Keyed Predictions pattern - **ML System Design**: Batch vs. online prediction serving tradeoffs, architecture patterns - **Apache Spark MLlib**: Distributed batch prediction at scale, best practices - **AWS SageMaker Batch Transform**: Managed batch inference service, cost optimization