メニュー
SpikeDecoder - Fully SNN-based implementation of Transformer decoder block for NLP applications, achieving 87-93% energy reduction while maintaining performance
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/hiyenwong/ai_collection --skill spikedecoder-snn-gpt-architectureコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SKILL.md を表示中
SOC 職業分類に基づく
| name | spikedecoder-snn-gpt-architecture |
| description | SpikeDecoder - Fully SNN-based implementation of Transformer decoder block for NLP applications, achieving 87-93% energy reduction while maintaining performance |
| version | 1.0.0 |
| category | spiking-neural-networks |
| activation_keywords | ["spiking neural network","SNN transformer","energy-efficient NLP","spike-based language model","neuromorphic computing","spike embedding","GPT SNN","spiking decoder"] |
| trigger_pattern | SNN transformer|spiking GPT|energy-efficient NLP|spike decoder|neuromorphic language model |
| authors | ["Claas Beger","Florian Walter","Alois Knoll"] |
| arxiv_id | 2606.12287 |
| published_date | 2026-06-10T00:00:00.000Z |
arXiv: 2606.12287 | Published: 2026-06-10 | Categories: cs.NE, cs.AI
The Transformer architecture faces critical challenges for deployment:
SpikeDecoder: First fully SNN-based implementation of the Transformer decoder block for natural language processing that:
[Token Embedding] → [Spike Embedding Layer] → [Spike Position Encoding]
↓
[Spiking Self-Attention]
↓
[Spike Residual Connection]
↓
[Spiking Feed-Forward Network]
↓
[Spike Layer Normalization]
↓
[Output Spike Pattern]
class SpikeEmbedding(nn.Module):
"""
Project text data into spike patterns.
Three embedding strategies tested:
- Rate coding: Frequency-based spike representation
- Temporal coding: Spike timing encodes information
- Population coding: Distributed spike patterns across neurons
"""
def __init__(
self,
vocab_size,
embed_dim,
spike_neurons_per_token=,
embedding_type=
):
().__init__()
.embedding_type = embedding_type
.spike_neurons_per_token = spike_neurons_per_token
.base_embedding = nn.Embedding(vocab_size, embed_dim)
embedding_type == :
.spike_rate_encoder = RateCodingEncoder(embed_dim)
embedding_type == :
.temporal_encoder = TemporalCodingEncoder(embed_dim)
embedding_type == :
.population_encoder = PopulationCodingEncoder(
embed_dim, spike_neurons_per_token
)
():
embedding = .base_embedding(token_ids)
.embedding_type == :
spike_pattern = .spike_rate_encoder(embedding, simulation_time)
.embedding_type == :
spike_pattern = .temporal_encoder(embedding, simulation_time)
.embedding_type == :
spike_pattern = .population_encoder(embedding, simulation_time)
spike_pattern
class SpikingSelfAttention(nn.Module):
"""
Spiking implementation of self-attention.
Key differences from ANN attention:
- Spike-based query/key/value computation
- Membrane potential dynamics for attention weights
- No softmax exponentiation (uses spike accumulation)
"""
def __init__(
self,
embed_dim,
num_heads,
tau_m=20.0, # membrane time constant
tau_s=5.0, # synaptic time constant
threshold=1.0
):
super().__init__()
self.embed_dim = embed_dim
self.num_heads = num_heads
# Spiking neurons for Q, K, V projections
self.q_neurons = SpikingNeuronLayer(embed_dim, tau_m, threshold)
self.k_neurons = SpikingNeuronLayer(embed_dim, tau_m, threshold)
self.v_neurons = SpikingNeuronLayer(embed_dim, tau_m, threshold)
# Synaptic dynamics
self.synapse_decay = nn.Parameter(torch.tensor(1.0 / tau_s))
def forward(self, spike_input, simulation_time):
"""
Compute spike-based self-attention.
Args:
- spike_input: [batch, seq_len, embed_dim, time]
- simulation_time: total timesteps
Returns:
- attention_spikes: [batch, seq_len, embed_dim, time]
"""
batch, seq_len, embed_dim, time = spike_input.shape
# Generate Q, K, V spike patterns
q_spikes = self.q_neurons(spike_input)
k_spikes = self.k_neurons(spike_input)
v_spikes = self.v_neurons(spike_input)
# Compute spike-based attention weights
# Accumulated spike coincidence replaces softmax
attention_weights = self.compute_spike_attention(q_spikes, k_spikes)
# Apply attention to V spikes
attention_spikes = self.apply_attention(attention_weights, v_spikes)
return attention_spikes
def compute_spike_attention(self, q_spikes, k_spikes):
"""
Compute attention weights from spike coincidence.
Key insight: Spike timing correlation approximates attention scores
without expensive softmax computation.
"""
# Spike coincidence matrix
coincidence = torch.bmm(
q_spikes.sum(dim=-1), # aggregate over time
k_spikes.sum(dim=-1).transpose(1, 2)
)
# Normalize via synaptic dynamics (not softmax)
# Use exponential synaptic decay for normalization
weights = coincidence * torch.exp(-self.synapse_decay)
return weights
def apply_attention(self, weights, v_spikes):
"""
Apply attention weights to value spikes.
"""
# Weighted spike accumulation
attended = torch.bmm(weights, v_spikes.sum(dim=-1))
# Regenerate spike pattern from attended values
return attended.unsqueeze(-1).expand(-1, -1, -1, time)
class SpikeLayerNorm(nn.Module):
"""
Layer normalization for spike patterns.
Challenge: Standard LayerNorm assumes continuous values.
Solution: Normalize membrane potentials before spike generation.
"""
def __init__(self, embed_dim, eps=1e-6):
super().__init__()
self.weight = nn.Parameter(torch.ones(embed_dim))
self.bias = nn.Parameter(torch.zeros(embed_dim))
self.eps = eps
def forward(self, membrane_potential):
"""
Normalize membrane potentials.
Args:
- membrane_potential: [batch, seq_len, embed_dim]
Returns:
- normalized_potential: normalized membrane state
"""
# Compute statistics on membrane potential (pre-spike)
mean = membrane_potential.mean(dim=-1, keepdim=True)
std = membrane_potential.std(dim=-1, keepdim=True) + self.eps
# Normalize and scale
normalized = (membrane_potential - mean) / std
normalized = normalized * self.weight + self.bias
return normalized
class SpikeResidualConnection(nn.Module):
"""
Residual connections for spiking networks.
Challenge: Direct addition of spike patterns may not preserve information.
Solution: Membrane potential integration before spike generation.
"""
def __init__(self, tau_m=20.0):
super().__init__()
self.tau_m = tau_m
def forward(self, input_spikes, output_spikes):
"""
Implement residual connection for spikes.
Key insight: Integrate membrane potentials from both pathways,
then generate unified spike output.
"""
# Convert spikes to membrane potentials
input_potential = self.spike_to_potential(input_spikes)
output_potential = self.spike_to_potential(output_spikes)
# Residual integration
residual_potential = input_potential + output_potential
# Generate spike output from integrated potential
residual_spikes = self.potential_to_spike(residual_potential)
return residual_spikes
def spike_to_potential(self, spikes):
"""
Convert spike pattern to membrane potential.
"""
# Exponential integration
potential = torch.zeros_like(spikes[..., 0])
for t in range(spikes.shape[-1]):
spike_t = spikes[..., t]
potential = potential * torch.exp(-1.0/self.tau_m) + spike_t
return potential
def potential_to_spike(self, potential, threshold=1.0):
"""
Generate spikes from membrane potential.
"""
spikes = (potential > threshold).float()
return spikes
def train_spikedecoder(
model,
train_dataset,
vocab,
num_epochs=50,
learning_rate=1e-3,
simulation_time=100
):
"""
Train SpikeDecoder directly (no ANN conversion needed).
Key training innovations:
- Surrogate gradient for non-differentiable spike function
- Spike-based loss functions
- Temporal backpropagation through time (BPTT)
"""
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
# Surrogate gradient function (for spike non-differentiability)
surrogate_grad = FastSigmoidSurrogate(alpha=10.0)
for epoch in range(num_epochs):
for batch in train_dataset:
input_ids = batch['input_ids']
target_ids = batch['target_ids']
# Forward pass with spike simulation
output_spikes = model(
input_ids,
simulation_time=simulation_time
)
# Decode output spikes to tokens
output_ids = decode_spikes_to_tokens(output_spikes, vocab)
# Cross-entropy loss on decoded tokens
loss = F.cross_entropy(output_ids, target_ids)
# Backward with surrogate gradients
optimizer.zero_grad()
loss.backward()
# Apply surrogate gradient to spike layers
apply_surrogate_gradient(model, surrogate_grad)
optimizer.step()
| Component | ANN Operations | SNN Operations | Energy Reduction |
|---|---|---|---|
| Attention | Softmax + MatMul | Spike Accumulation | ~90% |
| FFN | Dense + Activation | Spike Integration | ~85% |
| Normalization | Mean/Std Calculation | Membrane Decay | ~80% |
| Overall | Baseline | Spiking | 87-93% |
| Task | ANN Baseline | SpikeDecoder | Performance Gap |
|---|---|---|---|
| Language Modeling (PPL) | X | Y | +Z PPL |
| Text Generation | Coherent | Mostly Coherent | Minor artifacts |
| Memory Efficiency | High | Very Low | ~90% reduction |
class SpikeDecoder(nn.Module):
"""
Complete SNN-based Transformer decoder for NLP.
Args:
- vocab_size: number of tokens
- embed_dim: embedding dimension
- num_heads: attention heads
- num_layers: decoder layers
- spike_simulation_time: timesteps per forward pass
"""
def __init__(
self,
vocab_size=50000,
embed_dim=512,
num_heads=8,
num_layers=6,
spike_simulation_time=100
):
super().__init__()
self.spike_simulation_time = spike_simulation_time
# Spike embedding layer
self.spike_embedding = SpikeEmbedding(
vocab_size=vocab_size,
embed_dim=embed_dim,
embedding_type='rate' # best performance
)
# Spike position encoding
self.position_encoder = SpikePositionEncoder(embed_dim)
# Decoder layers
self.layers = nn.ModuleList([
SpikeDecoderLayer(embed_dim, num_heads)
for _ in range(num_layers)
])
# Output projection
self.output_projection = SpikeOutputProjection(embed_dim, vocab_size)
def forward(self, input_ids):
"""
Generate spike-based language model output.
Args:
- input_ids: [batch, seq_len]
Returns:
- output_logits: [batch, seq_len, vocab_size]
"""
batch, seq_len = input_ids.shape
# Embed tokens as spikes
spike_pattern = self.spike_embedding(
input_ids,
self.spike_simulation_time
)
# Add position encoding
spike_pattern = self.position_encoder(spike_pattern)
# Process through decoder layers
for layer in self.layers:
spike_pattern = layer(spike_pattern, self.spike_simulation_time)
# Decode to output logits
output_logits = self.output_projection(spike_pattern)
return output_logits
| Method | Task | Architecture | Training | Energy Reduction |
|---|---|---|---|---|
| SpikeBERT | Vision | Encoder-only | Conversion | ~80% |
| SpikeViT | Vision | Encoder-only | Conversion | ~85% |
| SpikeDecoder | NLP | Decoder | Direct | 87-93% |
Key Advantages:
@article{beger2026spikedecoder,
title={SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks},
author={Beger, Claas and Walter, Florian and Knoll, Alois},
journal={arXiv preprint arXiv:2606.12287},
year={2026}
}