| name | msft-mixture-overfitting |
| title | mSFT: Ranked Findings on Supervised Fine-Tuning Dataset Mixture Overfitting |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.21606 |
| keywords | ["Training","Dataset Mixture","Overfitting","Fine-Tuning","Empiricism"] |
| description | Identify three ranked findings on multi-task SFT: (1) heterogeneous overfitting—sub-datasets peak at different training points (contradicts uniform duration practice); (2) parameter divergence—excluding 1/10 of data shifts optimal points 0.91 epochs for remaining tasks; (3) SFT compute negligible (0.01% of training). Implement mSFT: iterative roll-out/roll-back search per-dataset. Robust across 0.5B-8B models, 9K-27K samples, 5-15 tasks, achieving +3.4% improvement with reduced FLOPs. |
Ranked Findings
Finding #1: Heterogeneous Overfitting Dynamics (Primary)
Discovery: Individual sub-datasets within multi-task mixtures reach peak performance at substantially different training points, contradicting industry standard practice of uniform training duration.
Evidence:
Figure 2 demonstrates across all tested configurations that different sub-datasets achieve maximum validation accuracy at different compute levels. This heterogeneity is systematic, not noise—consistent across model scales and task granularities.
Implication:
Standard practice of training all tasks for identical duration is suboptimal. Each task has a distinct "peak performance point" determined by:
- Task difficulty (harder tasks may need more compute)
- Dataset size (smaller datasets overfit earlier)
- Domain relatedness (related tasks may peak together)
Decision Impact:
Stopping uniformly at any point necessarily leaves some tasks undertrained (below peak) while others are overtrained (past peak). Heterogeneous stopping can recover performance lost to this tradeoff.
Finding #2: Parameter Divergence Problem (Secondary)
Discovery: When excluding even small fractions of training data (1/10), optimal stopping points for remaining datasets shift significantly. Naive approaches that pre-compute stopping points from an initial search then exclude datasets fundamentally fail.
Evidence:
Empirical measurements show:
- Excluding 1/10 of data shifts average optimal stopping point by 0.91 epochs
- Shifts are non-uniform (some tasks shift +2 epochs, others -1)
- Shifting parameters in either direction reduces performance on remaining tasks
Root Cause:
The optimization landscape couples all datasets through shared parameters. Removing one dataset creates new local optima for remaining tasks—the coupling cannot be separated.
Implication:
Cannot pre-compute optimal stopping per-dataset once, then use those points sequentially. Must search iteratively as datasets are modified.
Finding #3: SFT Compute Negligible (Enabling Condition)
Discovery: SFT (supervised fine-tuning) stage accounts for only 0.01% of total pre-training compute, making exhaustive per-dataset optimization computationally justified.
Arithmetic:
- Total pre-training compute: ~100 GPU-years
- SFT stage: ~0.01 GPU-years
- Even 100× search overhead in SFT remains <1% total training
