elastic-spiking-transformer

name

elastic-spiking-transformer

description

Matryoshka-style elasticity for Spiking Transformers - runtime-adaptive architecture enabling dynamic width and attention head slicing at inference without retraining. Applies to SNN deployment on neuromorphic hardware, edge AI, and adaptive computation. Activation: elastic spiking transformer, matryoshka spiking, runtime adaptive SNN, granularity-aware weight sharing, dynamic slicing spiking neural network.

Elastic Spiking Transformer

Runtime-adaptive Spiking Transformer architecture using Matryoshka-style nested elasticity for dynamic inference-time width/attention head adjustment without retraining, enabling efficient deployment across neuromorphic hardware with varying memory budgets.

Metadata

• Source: arXiv:2605.13869
• Authors: Alberto Ancilotto, Gianluca Amprimo, Stefano Di Carlo, Elisabetta Farella
• Published: 2026-05-04
• Categories: cs.NE, cs.AI, cs.CV

Core Methodology

Key Innovation

Current Spiking Transformers are rigid — trained and deployed as static networks with fixed parameter counts and computational graphs. This limits deployment on neuromorphic hardware (Loihi, SpiNNaker) where on-chip constraints require smaller models. The Elastic Spiking Transformer introduces Matryoshka-style nested elasticity into the spiking paradigm, enabling a single universal model to dynamically adjust its footprint at inference time.

Technical Framework

1. Granularity-Aware Weight Sharing

• Embeds nested elasticity across three core blocks:
  • Feature Extractor: Hierarchical width slicing
  • Spiking Self-Attention (SSA): Dynamic attention head reduction
  • Feed-Forward (FF) blocks: Nested layer width adjustment
• Single training produces a "universal model" containing sub-models of all sizes
• At inference, select sub-model by choosing width/head count — no retraining needed

2. Elasticity in SNNs — Unique Benefits

Unlike ANNs where elasticity only saves FLOPs, in SNNs elasticity provides dual energy savings:

• Parameter footprint reduction: Adjust to hardware memory budgets
• Spike firing rate reduction: Fewer active neurons → fewer spikes → proportional reduction in synaptic operations (SOPs)
• This dual benefit is unique to the event-driven spiking computation model

3. Training Protocol

• Train full-width model with standard spiking transformer training
• Loss includes contributions from multiple granularity levels (nested training signal)
• Weight sharing across all levels ensures sub-models are independently functional
• No separate training per configuration — one model, many deployment profiles

Code Example

# Conceptual Elastic Spiking Transformer
class ElasticSpikingTransformer(nn.Module):
    def __init__(self, max_embed_dim=512, max_heads=8, max_ff_dim=2048, 
                 num_layers=4, num_classes=10):
        super().__init__()
        # Nested elasticity: weights support all sub-configurations
        self.max_embed = max_embed_dim
        self.max_heads = max_heads
        self.max_ff = max_ff_dim
        
        # Feature extractor with granularity-aware weights
        self.feature_extractor = ElasticFeatureExtractor(
            max_dim=max_embed_dim
        )
        
        # Elastic spiking self-attention
        self.layers = nn.ModuleList([
            ElasticSpikingSelfAttention(
                embed_dim=max_embed_dim,
                max_heads=max_heads,
                ff_dim=max_ff_dim
            ) for _ in range(num_layers)
        ])
        self.head = nn.Linear(max_embed_dim, num_classes)
    
    def forward(self, x, target_width=None, target_heads=None, target_ff=None):
        """
        Runtime-adaptive inference: select sub-model configuration.
        If no config specified, use full model.
        """
        x = self.feature_extractor(x, target_dim=target_width)
        for layer in self.layers:
            x = layer(x, target_heads=target_heads, target_ff=target_ff)
        # Slice classifier output to match target width
        out = .head(x)
         target_width:
            out = out[:, :target_width]
         out


model = ElasticSpikingTransformer()


out_full = model(x)


out_edge = model(x, target_width=, target_heads=, target_ff=)


out_ultra = model(x, target_width=, target_heads=, target_ff=)

Applications

• Neuromorphic hardware deployment: Loihi, SpiNNaker, TrueNorth with varying on-chip memory
• Edge AI: Adaptive inference based on device capability and battery state
• Clinical gesture understanding: EHWGesture dataset, real-time adaptive processing
• Dynamic vision sensors: CIFAR10-DVS event-based vision with elastic compute
• Resource-constrained robotics: Adjust computation to available energy budget

Implementation Considerations

• Spiking-specific training: Ensure surrogate gradients work correctly with nested weight sharing
• Spike threshold tuning: Different sub-models may need different threshold configurations
• Temporal dynamics: Elasticity affects spike timing patterns — verify temporal coding integrity
• Memory vs. compute trade-off: Weight sharing requires storing full model; inference-time slicing saves compute

Pitfalls

• Sub-model accuracy degradation: Smaller configurations may lose critical spike patterns
• Temporal coding disruption: Reducing dimensions may alter spike timing relationships
• Hardware-specific optimization: General elasticity may not match neuromorphic chip constraints precisely
• Training instability: Nested loss signals from multiple granularity levels can conflict

Related Skills

• adaptive-spiking-transformer-energy-efficiency
• spiking-transformer-energy-efficiency
• wta-spiking-transformer-language
• spiking-transformer-unification
• quantized-snn-hardware-optimization