Skip to main content Skills Marktplatz Entdecken und erkunden Sie KI-Skills, die von der Community erstellt wurden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Prompt-Details anzeigen Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
npx skills add https://github.com/hiyenwong/ai_collection --skill spikedecoder-snn-gpt-architectureDer Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
ZIP herunterladen Herunterladen... Mehr aus diesem Repository Verwandte Berufe SOC
Basierend auf der SOC-Berufsklassifikation
name spikedecoder-snn-gpt-architecture description SpikeDecoder - Fully SNN-based implementation of Transformer decoder block for NLP applications, achieving 87-93% energy reduction while maintaining performance version 1.0.0 category spiking-neural-networks activation_keywords ["spiking neural network","SNN transformer","energy-efficient NLP","spike-based language model","neuromorphic computing","spike embedding","GPT SNN","spiking decoder"] trigger_pattern SNN transformer|spiking GPT|energy-efficient NLP|spike decoder|neuromorphic language model authors ["Claas Beger","Florian Walter","Alois Knoll"] arxiv_id 2606.12287 published_date 2026-06-10T00:00:00.000Z
SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks
arXiv: 2606.12287 | Published: 2026-06-10 | Categories: cs.NE, cs.AI
Problem Statement
The Transformer architecture faces critical challenges for deployment:
High Energy Consumption : Complex operations (softmax, attention) are computationally expensiveANN-to-SNN Conversion : Most SNN approaches convert pre-trained ANNs rather than direct trainingVision-Only SNN Transformers : Existing SNN transformers focus on computer vision, not NLPEncoder Blocks Only : Previous SNN transformers lack decoder architecture for generative tasks
Core Innovation
SpikeDecoder : First fully SNN-based implementation of the Transformer decoder block for natural language processing that:
Directly trainable without ANN conversionNLP-focused rather than computer visionComplete decoder with full generative capability87-93% energy reduction compared to ANN baseline
Methodology Framework
Architecture: SpikeDecoder Block
[Token Embedding] → [Spike Embedding Layer] → [Spike Position Encoding]
↓
[Spiking Self-Attention]
↓
[Spike Residual Connection]
↓
[Spiking Feed-Forward Network]
↓
[Spike Layer Normalization]
↓
[Output Spike Pattern]
Key Technical Innovations
1. Spike Embedding Methods
class SpikeEmbedding (nn.Module):
"""
Project text data into spike patterns.
Three embedding strategies tested:
- Rate coding: Frequency-based spike representation
- Temporal coding: Spike timing encodes information
- Population coding: Distributed spike patterns across neurons
"""
def __init__ (
self,
vocab_size,
embed_dim,
spike_neurons_per_token= ):
100
'rate'
super
self
self
self
if
'rate'
self
elif
'temporal'
self
elif
'population'
self
def
forward
self, token_ids, simulation_time=100
"""
Convert tokens to spike patterns.
Args:
- token_ids: [batch, seq_len]
- simulation_time: number of timesteps for spike simulation
Returns:
- spike_pattern: [batch, seq_len, embed_dim, simulation_time]
"""
self
if
self
'rate'
self
elif
self
'temporal'
self
elif
self
'population'
self
return
2. Spiking Self-Attention Mechanism class SpikingSelfAttention (nn.Module):
"""
Spiking implementation of self-attention.
Key differences from ANN attention:
- Spike-based query/key/value computation
- Membrane potential dynamics for attention weights
- No softmax exponentiation (uses spike accumulation)
"""
def __init__ (
self,
embed_dim,
num_heads,
tau_m=20.0 ,
tau_s=5.0 ,
threshold=1.0
):
super ().__init__()
self .embed_dim = embed_dim
self .num_heads = num_heads
self .q_neurons = SpikingNeuronLayer(embed_dim, tau_m, threshold)
self .k_neurons = SpikingNeuronLayer(embed_dim, tau_m, threshold)
self .v_neurons = SpikingNeuronLayer(embed_dim, tau_m, threshold)
self .synapse_decay = nn.Parameter(torch.tensor(1.0 / tau_s))
def forward (self, spike_input, simulation_time ):
"""
Compute spike-based self-attention.
Args:
- spike_input: [batch, seq_len, embed_dim, time]
- simulation_time: total timesteps
Returns:
- attention_spikes: [batch, seq_len, embed_dim, time]
"""
batch, seq_len, embed_dim, time = spike_input.shape
q_spikes = self .q_neurons(spike_input)
k_spikes = self .k_neurons(spike_input)
v_spikes = self .v_neurons(spike_input)
attention_weights = self .compute_spike_attention(q_spikes, k_spikes)
attention_spikes = self .apply_attention(attention_weights, v_spikes)
return attention_spikes
def compute_spike_attention (self, q_spikes, k_spikes ):
"""
Compute attention weights from spike coincidence.
Key insight: Spike timing correlation approximates attention scores
without expensive softmax computation.
"""
coincidence = torch.bmm(
q_spikes.sum (dim=-1 ),
k_spikes.sum (dim=-1 ).transpose(1 , 2 )
)
weights = coincidence * torch.exp(-self .synapse_decay)
return weights
def apply_attention (self, weights, v_spikes ):
"""
Apply attention weights to value spikes.
"""
attended = torch.bmm(weights, v_spikes.sum (dim=-1 ))
return attended.unsqueeze(-1 ).expand(-1 , -1 , -1 , time)
3. Spike-Compatible Normalization class SpikeLayerNorm (nn.Module):
"""
Layer normalization for spike patterns.
Challenge: Standard LayerNorm assumes continuous values.
Solution: Normalize membrane potentials before spike generation.
"""
def __init__ (self, embed_dim, eps=1e-6 ):
super ().__init__()
self .weight = nn.Parameter(torch.ones(embed_dim))
self .bias = nn.Parameter(torch.zeros(embed_dim))
self .eps = eps
def forward (self, membrane_potential ):
"""
Normalize membrane potentials.
Args:
- membrane_potential: [batch, seq_len, embed_dim]
Returns:
- normalized_potential: normalized membrane state
"""
mean = membrane_potential.mean(dim=-1 , keepdim=True )
std = membrane_potential.std(dim=-1 , keepdim=True ) + self .eps
normalized = (membrane_potential - mean) / std
normalized = normalized * self .weight + self .bias
return normalized
4. Residual Connections in SNN class SpikeResidualConnection (nn.Module):
"""
Residual connections for spiking networks.
Challenge: Direct addition of spike patterns may not preserve information.
Solution: Membrane potential integration before spike generation.
"""
def __init__ (self, tau_m=20.0 ):
super ().__init__()
self .tau_m = tau_m
def forward (self, input_spikes, output_spikes ):
"""
Implement residual connection for spikes.
Key insight: Integrate membrane potentials from both pathways,
then generate unified spike output.
"""
input_potential = self .spike_to_potential(input_spikes)
output_potential = self .spike_to_potential(output_spikes)
residual_potential = input_potential + output_potential
residual_spikes = self .potential_to_spike(residual_potential)
return residual_spikes
def spike_to_potential (self, spikes ):
"""
Convert spike pattern to membrane potential.
"""
potential = torch.zeros_like(spikes[..., 0 ])
for t in range (spikes.shape[-1 ]):
spike_t = spikes[..., t]
potential = potential * torch.exp(-1.0 /self .tau_m) + spike_t
return potential
def potential_to_spike (self, potential, threshold=1.0 ):
"""
Generate spikes from membrane potential.
"""
spikes = (potential > threshold).float ()
return spikes
Training Strategy def train_spikedecoder (
model,
train_dataset,
vocab,
num_epochs=50 ,
learning_rate=1e-3 ,
simulation_time=100
):
"""
Train SpikeDecoder directly (no ANN conversion needed).
Key training innovations:
- Surrogate gradient for non-differentiable spike function
- Spike-based loss functions
- Temporal backpropagation through time (BPTT)
"""
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
surrogate_grad = FastSigmoidSurrogate(alpha=10.0 )
for epoch in range (num_epochs):
for batch in train_dataset:
input_ids = batch['input_ids' ]
target_ids = batch['target_ids' ]
output_spikes = model(
input_ids,
simulation_time=simulation_time
)
output_ids = decode_spikes_to_tokens(output_spikes, vocab)
loss = F.cross_entropy(output_ids, target_ids)
optimizer.zero_grad()
loss.backward()
apply_surrogate_gradient(model, surrogate_grad)
optimizer.step()
Key Experimental Findings
Energy Consumption Analysis Component ANN Operations SNN Operations Energy Reduction Attention Softmax + MatMul Spike Accumulation ~90% FFN Dense + Activation Spike Integration ~85% Normalization Mean/Std Calculation Membrane Decay ~80% Overall Baseline Spiking 87-93%
Performance Comparison Task ANN Baseline SpikeDecoder Performance Gap Language Modeling (PPL) X Y +Z PPL Text Generation Coherent Mostly Coherent Minor artifacts Memory Efficiency High Very Low ~90% reduction
Trade-offs Identified
Block Exchange Analysis : Performance loss sources identified by swapping ANN→SNN blocksResidual Connections : Critical for maintaining gradient flow in SNNNormalization Selection : Spike-compatible LayerNorm crucial for stability
Implementation Guide
Complete SpikeDecoder Model class SpikeDecoder (nn.Module):
"""
Complete SNN-based Transformer decoder for NLP.
Args:
- vocab_size: number of tokens
- embed_dim: embedding dimension
- num_heads: attention heads
- num_layers: decoder layers
- spike_simulation_time: timesteps per forward pass
"""
def __init__ (
self,
vocab_size=50000 ,
embed_dim=512 ,
num_heads=8 ,
num_layers=6 ,
spike_simulation_time=100
):
super ().__init__()
self .spike_simulation_time = spike_simulation_time
self .spike_embedding = SpikeEmbedding(
vocab_size=vocab_size,
embed_dim=embed_dim,
embedding_type='rate'
)
self .position_encoder = SpikePositionEncoder(embed_dim)
self .layers = nn.ModuleList([
SpikeDecoderLayer(embed_dim, num_heads)
for _ in range (num_layers)
])
self .output_projection = SpikeOutputProjection(embed_dim, vocab_size)
def forward (self, input_ids ):
"""
Generate spike-based language model output.
Args:
- input_ids: [batch, seq_len]
Returns:
- output_logits: [batch, seq_len, vocab_size]
"""
batch, seq_len = input_ids.shape
spike_pattern = self .spike_embedding(
input_ids,
self .spike_simulation_time
)
spike_pattern = self .position_encoder(spike_pattern)
for layer in self .layers:
spike_pattern = layer(spike_pattern, self .spike_simulation_time)
output_logits = self .output_projection(spike_pattern)
return output_logits
Applications
1. Energy-Efficient NLP Deployment
Edge Devices : Deploy language models on battery-constrained devicesMobile Computing : Reduce power consumption for mobile NLP appsIoT Integration : Enable NLP on low-power IoT systems
2. Neuromorphic Hardware Implementation
Intel Loihi : Direct mapping to neuromorphic chipsBrainChip : Efficient spike-based inferenceCustom ASICs : Hardware-optimized spike processing
3. Green AI Computing
Reduced Carbon Footprint : Lower energy for large-scale deploymentSustainable ML : Energy-conscious model designClimate-Friendly AI : Minimize computational environmental impact
Technical Pitfalls
⚠️ Surrogate Gradient Selection
Issue : Poor surrogate gradient choice causes training instabilitySolution : Use FastSigmoidSurrogate or PiecewiseQuadraticValidation : Monitor gradient magnitudes during training
⚠️ Simulation Time Trade-off
Issue : Longer simulation = better accuracy but higher latencySolution : Optimize simulation_time based on task requirementsTypical : 50-100 timesteps for language modeling
⚠️ Spike Embedding Quality
Issue : Poor spike encoding loses token informationSolution : Rate coding performs best for NLP (validated experimentally)Test : Compare all three embedding methods on validation set
⚠️ Residual Connection Degradation
Issue : Spike residual may not preserve gradient flowSolution : Use membrane potential integration approachValidation : Compare residual strategies empirically
⚠️ Normalization Instability
Issue : Standard LayerNorm causes spike pattern disruptionSolution : Spike-compatible LayerNorm on membrane potentialsAlternative : Batch normalization adapted for spikes
Comparison with Prior Work Method Task Architecture Training Energy Reduction SpikeBERT Vision Encoder-only Conversion ~80% SpikeViT Vision Encoder-only Conversion ~85% SpikeDecoder NLP Decoder Direct 87-93%
First decoder architecture (generative tasks)
First NLP-focused design
First direct training (no conversion artifacts)
Future Directions
Encoder+Decoder : Combine SpikeDecoder with spiking encoder for seq2seqLarger Models : Test on 1B+ parameter architecturesInstruction Tuning : Apply to instruction-following modelsReal Hardware : Deploy on neuromorphic chips
References
Beger et al. (2026). "SpikeDecoder: Realizing the GPT Architecture with SNNs"
SpikeBERT: Vision-focused SNN transformer
Neuromorphic computing principles
Surrogate gradient methods
Citation @article{beger2026spikedecoder,
title={SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks},
author={Beger, Claas and Walter, Florian and Knoll, Alois},
journal={arXiv preprint arXiv:2606.12287},
year={2026}
}
