elastic-spiking-transformer-matryoshka

name

elastic-spiking-transformer-matryoshka

description

Matryoshka-style elastic Spiking Transformer with runtime-adaptive width and attention head slicing for deployment across hardware budgets without retraining. Reduces spike firing rates proportionally to parameter footprint. Use when deploying SNNs on constrained neuromorphic hardware, edge devices, or gesture recognition tasks. Activation: elastic spiking transformer, Matryoshka spiking network, runtime-adaptive SNN, dynamic width spiking, gesture understanding SNN, nested elasticity SNN

Elastic Spiking Transformer with Matryoshka-Style Nested Elasticity

Overview

Introduces runtime-adaptive architecture into Spiking Transformers via Matryoshka-style nested elasticity. A single universal model dynamically slices network width and attention heads at inference time without retraining, adapting to different hardware memory budgets. Reducing active neurons also lowers spike firing rates, yielding proportional reductions in synaptic operations — an energy benefit unique to SNNs (not available in standard ANNs).

Source: arXiv:2605.13869 — "Elastic Spiking Transformers for Efficient Gesture Understanding"

Core Mechanism

Granularity-Aware Weight Sharing

Instead of training multiple models for different hardware budgets, train one universal model with nested sub-networks:

Full model:    [████████████████████████████████]  (all neurons/heads active)
Sub-model 75%: [████████████████████████████    ]  (top 75% of weights)
Sub-model 50%: [████████████████                ]  (top 50% of weights)
Sub-model 25%: [████████                        ]  (top 25% of weights)

Three Elastic Components

1. Feature Extractor: Nested channel-width scaling
2. Spiking Self-Attention: Nested attention head slicing
3. Feed-Forward Block: Nested MLP width scaling

Key Insight for SNNs

In ANNs, reducing parameters only saves FLOPs. In SNNs, reducing active neurons also lowers spike firing rates, yielding proportional reductions in synaptic operations. This creates a dual energy saving mechanism unique to spiking architectures.

Implementation Steps

1. Train universal model with full capacity
2. Use Matryoshka-style loss: jointly optimize all sub-network widths during training
3. At inference, select sub-network matching hardware memory budget
4. Route input through selected sub-network (no retraining needed)
5. Monitor spike firing rates — they should scale proportionally with active parameters

Energy Scaling

Sub-network width	Parameter count	Spike rate	SynOps
100% (full)	1.0x	1.0x	1.0x
75%	0.75x	~0.7x	~0.55x
50%	0.5x	~0.45x	~0.25x
25%	0.25x	~0.2x	~0.05x

Pitfalls

1. Weight ordering: Must learn nested structure during training (use auxiliary losses for each sub-width)
2. Attention head slicing: Slice heads, not dimensions within heads — preserves attention mechanism semantics
3. Spike timing: Elastic width changes temporal dynamics — may need threshold recalibration per sub-network
4. Hardware mapping: Each sub-network maps to different neuromorphic chip configurations (Loihi, SpiNNaker)
5. Not all tasks tolerate elastic width: Temporal prediction tasks may need full capacity

Related Skills

• adaptive-spiking-transformer-energy-efficiency: Threshold-based spike sparsity (complementary)
• elastic-spiking-transformer: Elasticity via Matryoshka representation (this skill)

Activation Keywords

• elastic spiking transformer, Matryoshka spiking network, runtime-adaptive SNN, dynamic width spiking, gesture understanding SNN, nested elasticity SNN, energy-efficient edge SNN, Loihi deployment, SpiNNaker deployment