maestro-pruning-bad-experts-mixture-of-experts

name	maestro-pruning-bad-experts-mixture-of-experts
description	MAESTRO: Markov-chain Approximated Expert Sparsification via Transition-based Routing for MoE structured pruning. Models expert activation as Ergodic Markov chains for globally-aware importance. Outperforms baselines by 10.61% at 50% compression. Lower cross-task variance. Activation: mixture-of-experts, expert pruning, MoE deployment, structured pruning, language model efficiency.
metadata	{"arxiv_id":"2607.08601","published":"2026-07-09","authors":"Palaash Goel, Ayush Maheshwari, Tanmoy Chakraborty","tags":["mixture-of-experts","expert-pruning","MoE-deployment","structured-pruning","language-model-efficiency"]}

It Takes a MAESTRO To Prune Bad Experts

Overview

MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting) is a structured pruning framework for Mixture-of-Experts (MoE) architectures that models autoregressive expert activation trajectories as Ergodic Markov chains. This enables globally-aware importance assessment that captures cross-layer dependencies, unlike existing methods using locally derived heuristics.

Key Innovations

Markov Chain Expert Modeling

• Models autoregressive expert activation trajectories as Ergodic Markov chains
• Stationary distributions encode cross-layer dependencies
• Yields globally-aware importance heuristic
• Overcomes blindness of locally derived heuristics to routing interdependencies

Structured Pruning for MoE

• Specifically designed for MoE architectures (not adapted from dense transformer methods)
• Addresses the deployment bottleneck of full expert banks residing in memory
• Enables compression while maintaining performance across diverse domains

Generalization Performance

• Evaluated across five diverse domains including Safety, Bias, and Ethics
• Outperforms state-of-the-art baselines by up to 10.61% in average performance retention
• Under strict 50% compression regime
• Substantially lower cross-task variance — more consistent generalization

Methodology

1. Markov Chain Construction: Model expert activation as Ergodic Markov chain
2. Stationary Distribution: Compute stationary distributions encoding cross-layer dependencies
3. Importance Scoring: Rank experts by globally-aware importance heuristic
4. Pruning: Remove low-importance experts under target compression ratio
5. Evaluation: Test across diverse domains for performance and generalization

Implications

• Global routing-aware pruning is essential for MoE compression
• Markov chain modeling captures inter-layer dependencies missed by local methods
• Lower cross-task variance indicates more reliable deployment
• Enables MoE deployment in resource-constrained settings

Pitfalls

• Markov chain assumption of Ergodicity may not hold for all MoE architectures
• 50% compression may be too aggressive for some applications
• Five domains may not capture all relevant evaluation axes
• Stationary distribution computation adds preprocessing overhead

Activation Keywords

mixture-of-experts, expert pruning, MAESTRO, Markov chain, structured pruning, MoE deployment, routing-aware pruning, language model efficiency, cross-task generalization

Paper Reference

arXiv:2607.08601 - "It Takes a MAESTRO To Prune Bad Experts" (Jul 2026)