online-learning

# Online Learning Pattern ## Classification - **Domain**: Computer Science, AI/ML - **Category**: ML System Design Patterns - **Novelty**: 8/10 (advanced pattern for continuous adaptation) - **Practitioner Evidence**: 9/10 (Kafka-ML, research-backed, emerging production use) ## Mental Model Online learning (incremental learning) updates models continuously as new data arrives, one sample at a time or in mini-batches, without retraining from scratch. Like learning a language through daily conversations versus intensive courses—knowledge accumulates gradually through exposure, adapting to new patterns without forgetting fundamentals. ## When to Use - Data distribution shifts frequently (concept drift, user behavior changes) - Retraining entire model is too expensive or slow (large datasets, limited compute) - Fresh predictions critical (stock trading, recommendation systems, personalization) - Continuous labeled feedback available (user clicks, transaction outcomes, sensor readings) - Model must adapt to new classes/patterns without catastrophic forgetting ## Core Framework ### 1. Learning Scenario Selection **Choose appropriate incremental learning paradigm** **Domain-Incremental Learning (DI)**: - Data distribution changes over time but task stays same - Example: Spam detection adapting to new spam tactics - Use when: Input patterns evolve but output categories fixed **Task-Incremental Learning (TI)**: - Multiple related tasks learned sequentially - Example: Multi-language translation learned one language pair at a time - Use when: Related tasks arrive incrementally, task ID known at test time **Class-Incremental Learning (CI)**: - New output classes added over time - Example: Product categorization with new product types added monthly - Use when: Output space grows, must classify new + old classes without task ID ### 2. Catastrophic Forgetting Prevention **Prevent performance degradation on old data when learning new patterns** **Regularization-Based Methods**: - Elastic Weight Consolidation (EWC): Penalize changes to important weights (Fisher information) - Learning without Forgetting (LwF): Preserve old predictions via knowledge distillation - Synaptic Intelligence: Track weight importance during learning, protect critical weights **Replay-Based Methods**: - Experience Replay: Store subset of old examples, mix with new data during updates - Generative Replay: Use generative model to synthesize old data patterns for rehearsal - Hybrid: Combine small memory buffer (1-5% of data) with regularization **Architecture-Based Methods**: - Progressive Neural Networks: Add new sub-networks for new tasks, freeze old ones - Dynamic Expandable Representation: Grow model capacity selectively for new patterns ### 3. Data Stream Processing Architecture **Configure infrastructure for continuous learning** - Stream data from Kafka/Kinesis topics (labeled_examples, user_feedback) - Implement sliding window for mini-batch updates (100-1000 samples per update) - Use stateful stream processing (Flink, Kafka Streams) for aggregating gradients - Checkpoint model state periodically (every N updates) for fault tolerance ### 4. Incremental Update Algorithm **Apply efficient gradient updates without full retraining** **Stochastic Gradient Descent (SGD) Variants**: - Process each example: compute loss → gradient → update weights - Use adaptive learning rates (AdaGrad, RMSprop, Adam) for stability - Decay learning rate over time (prevent oscillation as knowledge accumulates) **Mini-Batch Updates**: - Accumulate 50-500 examples → compute average gradient → update - Balance: Larger batches = stable updates, smaller batches = faster adaptation - Use gradient clipping to prevent exploding gradients from outliers **Second-Order Methods** (for shallow models): - Online Newton Step, Online AROW for linear/logistic models - More sample-efficient but higher computational cost per update ### 5. Model Evaluation & Drift Detection **Monitor performance and detect when retraining needed** - Track metrics on validation stream (separate from training stream) - Detect drift: Compare recent performance vs. baseline (sliding window metrics) - Use statistical tests (Kolmogorov-Smirnov, Page-Hinkley) for distribution shift - Trigger full retraining if online updates can't recover performance ### 6. Hyperparameter Adaptation **Adjust learning configuration based on stream characteristics** - Start with higher learning rate (faster initial adaptation), decay over time - Increase batch size as data accumulates (more stable updates with more data) - Adjust regularization strength based on forgetting rate (EWC lambda parameter) - Use meta-learning to tune hyperparameters online (learn learning rate schedules) ### 7. Production Deployment Strategy **Safely deploy continuously updating models** - Shadow mode: Run online learner alongside static model, compare predictions - Canary deployment: Route 5-10% traffic to online model, monitor metrics - A/B testing: Compare online learning vs. periodic batch retraining - Rollback mechanism: Revert to previous checkpoint if performance degrades ## Practical Application ### Personalized News Recommendations **Problem**: User interests change rapidly, daily retraining too slow **Online Learning Solution**: 1. User interactions stream to Kafka (user_id, article_id, click, timestamp) 2. Flink aggregates interactions into mini-batches (500 examples per 30 seconds) 3. Neural collaborative filtering model updates via Adam optimizer (lr=0.001, decay=0.999) 4. Experience replay: Buffer 10K recent examples, mix 20% old + 80% new per batch 5. Track click-through rate per user segment, rollback if drops >5% from baseline **Result**: 15% CTR improvement vs. daily batch retraining, 2-hour adaptation to trending topics ### Fraud Detection with Evolving Tactics **Problem**: Fraudsters adapt tactics weekly, batch models lag behind **Online Learning Solution**: 1. Transaction outcomes stream with 24-hour label delay (fraud confirmed/denied) 2. Gradient boosting model (XGBoost) with incremental updates (learning_rate=0.05) 3. Store 5K recent fraud examples in memory buffer for replay (prevent forgetting fraud patterns) 4. Page-Hinkley test monitors false positive rate (alert if statistically significant spike) 5. Full retraining triggered monthly or when drift detector fires **Result**: 40% faster detection of new fraud patterns, 8% reduction in false positives ### Product Categorization with New Product Types **Problem**: New product categories added monthly (CI scenario) **Online Learning Solution**: 1. New products labeled by ops team, streamed to training pipeline 2. Dynamic class-incremental learning: Add output neurons for new categories 3. Knowledge distillation: Freeze old predictions, only update for new classes 4. Balanced sampling: Equal representation of new classes + old classes in mini-batches 5. Evaluate on held-out old classes to detect catastrophic forgetting (>2% drop triggers intervention) **Result**: Support 50 new categories/month without full retraining, maintain 95% accuracy on old classes ## Edge Cases & Nuances **Label Delay**: Feedback arrives hours/days after prediction - Use delayed reward learning: Buffer predictions, apply updates when labels arrive - Implement temporal credit assignment (which prediction led to outcome?) - Consider imbalanced delayed feedback (positive outcomes reported faster than negative) **Outlier Robustness**: Adversarial examples or noise in stream - Use robust loss functions (Huber loss instead of MSE, focal loss for class imbalance) - Implement anomaly detection filter before model updates (flag suspicious examples) - Apply gradient clipping (cap gradient magnitude at 1.0-10.0) **Cold Start for New Entities**: New users/items without history - Initialize embeddings with content-based features or cluster averages - Use meta-learning (MAML) for fast adaptation with few examples - Fallback to population statistics until entity-specific data accumulates **Memory Constraints**: Limited storage for replay buffers - Prioritize examples for replay (reservoir sampling, importance weighting) - Use coreset construction: Select representative subset of old data - Compress experiences via generative model (GAN, VAE) for synthetic replay ## Anti-Patterns **No Forgetting Prevention**: Pure SGD on new data, forgetting old patterns within hours **Ignoring Data Distribution Shifts**: Blindly updating without drift detection or evaluation **Over-Aggressive Learning Rates**: High LR causing oscillation and catastrophic forgetting **No Rollback Strategy**: Deploying continuously updating model without safety nets ## Trade-offs **Online Learning vs. Batch Retraining**: - Online: Continuous adaptation, low latency updates, risk of drift/instability - Batch: Stable performance, expensive retraining, lag in adaptation **Replay Buffer Size**: - Larger (10% of data): Better retention, higher memory cost, slower updates - Smaller (1% of data): Memory-efficient, faster updates, more forgetting risk **Update Frequency**: - High (every 100 examples): Fast adaptation, potential instability, high compute - Low (every 10K examples): Stable updates, slower adaptation, bursty resource usage ## Related Frameworks - **Streaming Inference Pattern**: Real-time predictions on streaming data (inference side) - **Batch Processing Pattern**: Full retraining periodically (alternative to online learning) - **Continual Learning**: Broader field including task-incremental, class-incremental scenarios - **Transfer Learning**: Pre-train on large dataset, fine-tune on specific task (related adaptation strategy) - **Active Learning**: Select most informative examples for labeling (complement to online learning) ## Practitioner Sources - **Kafka-ML Framework**: Online learning infrastructure with Kafka + TensorFlow/PyTorch - **IBM Continual Learning**: Survey of methods, catastrophic forgetting solutions - **Nature Machine Intelligence** (2022): Three types of incremental learning taxonomy - **Flink ML**: Apache Flink for online model training and drift detection - **Chip Huyen - ML Systems Design**: Online learning in production systems, best practices