| name | partial-fusion-neural-networks |
| title | Partial Fusion of Neural Networks |
| description | Partial fusion of neural networks interpolating between ensembles and weight aggregation via neuron-level similarity matching and partial optimal transport. Frames partial fusion as generalized pruning where neurons are deleted or linearly combined. |
| arxiv | 2605.22350 |
| authors | ["Fabian Morelli","Stephan Eckstein"] |
| published | 2026-05-21 |
| tags | ["model merging","ensembles","weight aggregation","partial optimal transport","neuron matching","pruning"] |
| category | ai_collection |
Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation
arXiv: 2605.22350
Summary
Partial fusion of networks interpolates between two extremes: ensembles (high computational cost, high accuracy) and weight aggregation (low cost, lower accuracy). The core idea extends weight aggregation by leveraging neuron-level similarity between networks — only the weights of the most similar neurons are aggregated, while dissimilar neurons are kept separate.
Key Concepts
1. Neuron Matching via Partial Optimal Transport
- Networks are aligned by computing neuron similarity across models using partial optimal transport
- Unlike standard weight averaging (which averages all corresponding weights), partial fusion only averages weights for matched neuron pairs above a similarity threshold
- Unmatched/dissimilar neurons are retained individually, preserving diversity
2. Weight Aggregation ↔ Ensemble Interpolation
- Pure ensemble: All model outputs are combined (keep all neurons separate)
- Full weight aggregation: All corresponding weights are averaged (remove all neuron diversity)
- Partial fusion: A tunable middle ground — some neurons are aggregated, others are kept separate
3. Partial Fusion as Generalized Pruning of Ensembles
- The paper frames weight aggregation and partial fusion as a form of generalized pruning of ensemble models
- In traditional pruning, neurons are deleted; in partial fusion, neurons can be either deleted OR linearly combined
- This provides a smoother, more flexible accuracy–cost tradeoff than standard pruning
Method
- Train multiple networks (the ensemble members) on the same task
- Compute neuron similarity across networks (e.g., using activation correlations, weight similarity, or representation similarity)
- Solve a partial optimal transport problem to find matched neuron pairs and decide which neurons to fuse vs. keep separate
- Fuse matched neurons by averaging their weights (or some convex combination)
- Keep unmatched neurons as separate computational paths
- Optionally prune low-salience fused neurons
Use Cases
