| name | chanrg-rna-structure-generalization |
| title | Paradigm Challenge: RNA Structure Prediction Generalization Fails with Foundation Models |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.22330 |
| keywords | ["RNA Structure Prediction","Generalization Failure","Paradigm Challenge","Out-of-Distribution","Foundation Models"] |
| description | Overturn the assumption that scaling foundation models improves RNA structure prediction by understanding why they fail out-of-distribution. Includes structure-aware deduplication revealing 33-fold residual redundancy in prior benchmarks, out-of-distribution test regimes (GenA, GenC, GenF), and root cause analysis showing coverage and wiring failures. Foundation models achieving 67.3% on held-out test drop to 18.0% OOD (26.7% retention), while structured decoders retain 92.3%, enabling practitioners to recognize when scaling fails and when inductive structure matters. |
| category | Paradigm Challenge |
Prior Belief
The deep learning community widely believed that foundation models represent the frontier of RNA structure prediction. The intuition: scaling parameters, training data, and compute should overcome domain challenges through learned representations. Evidence supporting this belief included strong performance on standard benchmarks (e.g., Archive of Structural Proteomic Data) where foundation models achieved the highest held-out test accuracy. Practitioners made resource allocation decisions based on this assumption: invest in larger models, more pretraining, and expect generalization.
The Falsifying Experiment
CHANRG introduces structure-aware deduplication and out-of-distribution test regimes that overturn this assumption.
Core Experiment Protocol: Using Rfam clustering, the authors deduplicated 10+ million sequences by structural similarity, discovering that prior "held-out" test sets contained 33-fold residual structural redundancy. The same RNA structures, represented by different sequences, leaked across train-test boundaries. This violates the generalization premise: held-out accuracy measures interpolation within the training structure distribution, not true generalization.
Out-of-Distribution Regimes: Three complementary OOD splits test different forms of structural transfer:
- GenA: Architecturally distant RNAs (different secondary structure families)
- GenC: Evolutionarily distant organisms (structural conserved function, sequence diverged)
- GenF: Phylogenetically distant lineages (maximum distance within family)
Measurement: Direct accuracy comparison on held-out (HD) vs each OOD regime, revealing performance cliff for foundation models.
Sample Size & Robustness: 170,083 non-redundant structures tested across multiple models, with length-controlled analysis confirming the effect persists independent of sequence length.
The Performance Inversion
Foundation models exhibit dramatic performance collapse:
- Held-out accuracy: 67.3%
- Out-of-distribution accuracy: 18.0%
- Retention ratio: 26.7% (worst generalization)
Contrast with structured prediction baselines:
- Structured decoders: 30.2% held-out → 27.8% OOD (92.3% retention)
- Direct neural predictors: 35.0% held-out → 28.8% OOD (82.5% retention)
The inversion is stark: the model achieving highest held-out performance generalizes least robustly. This contradicts the scaling paradigm's core promise.
