rft-visual-continual-learning

# Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning > Demonstrates that Reinforcement Fine-Tuning (RFT), specifically GRPO, is inherently more resilient to catastrophic forgetting than Supervised Fine-Tuning (SFT) in challenging visual continual learning settings like class-incremental learning. ## Metadata - **Source**: arXiv:2605.09640 - **Authors**: Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu - **Published**: 2026-05-10 ## Core Problem **Visual Continual Learning**: Models must learn to recognize new visual classes incrementally without forgetting previously learned classes. Class-incremental learning is particularly challenging as the model must distinguish between all seen classes, old and new. **Catastrophic Forgetting**: Standard fine-tuning on new class data causes significant performance degradation on old classes. ## Key Finding **RFT > SFT for Continual Learning**: Reinforcement Fine-Tuning (RFT), exemplified by GRPO, shows significantly better retention of old task performance compared to Supervised Fine-Tuning (SFT). ### Why RFT Forgets Less 1. **Reward-based learning**: RFT optimizes for task completion rewards rather than exact label matching, learning more generalizable representations. 2. **Implicit regularization**: The RL objective naturally regularizes updates, preventing overfitting to new class features. 3. **Policy gradient stability**: Unlike supervised loss gradients which can be sharp and task-specific, RL gradients tend to be smoother and more stable. 4. **Exploration bonus**: RL's inherent exploration encourages finding solutions that work across tasks rather than memorizing new ones. ## Methodology ### GRPO for Visual CL - Uses Group Relative Policy Optimization for visual classification tasks - Compares performance within groups of samples to compute advantage estimates - No value function needed, reducing training complexity ### Training Setup - Class-incremental protocol: sequentially introduce new classes - No replay buffer (or minimal buffer) - Evaluate on all classes seen so far after each increment ### Comparison Baselines - Standard SFT with cross-entropy loss - EWC (Elastic Weight Consolidation) - Replay-based methods - Other CL baselines ## Results **Key Finding**: RFT (GRPO) maintains higher accuracy on old classes while achieving comparable performance on new classes, resulting in better overall continual learning performance. ## Implementation Guide ### Prerequisites - Vision model (ViT, ResNet, etc.) - GRPO implementation (TRL or similar) - Class-incremental dataset setup ### Step-by-Step 1. **Define reward function**: Classification accuracy or confidence-based rewards 2. **Group formation**: Group samples for relative policy optimization 3. **Policy optimization**: Apply GRPO updates without explicit value function 4. **Sequential training**: Incrementally add new classes and continue training ### Code Sketch ```python # Reward for classification def compute_rewards(predictions, targets): # Positive reward for correct classification # Scaled by confidence correct = (predictions == targets).float() return correct * confidence_scale # GRPO-style update advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8) policy_loss = -(advantages * log_prob_ratio).mean() ``` ## Applications - Visual class-incremental learning without replay - Medical image classification with new disease classes - Autonomous vehicle perception with new object types - Industrial defect detection with new defect categories ## Pitfalls - **Reward design**: Poor reward functions can lead to suboptimal policies - **Sample efficiency**: RFT typically requires more samples than SFT - **Training stability**: RL training can be less stable than supervised training - **Scalability**: May not scale as well as SFT for very large class sets ## Related Skills - zeroth-order-adaptation-forgetting-theory - catastrophic-forgetting-mitigation - reinforcement-learning-fine-tuning