dendritic-in-context-learning-snn

name

dendritic-in-context-learning-snn

description

Dendritic In-Context Learning (DendriCL): a single-layer compartmental spiking neural network whose apical/dendritic subthreshold dynamics implement online Widrow-Hoff LMS, giving in-context learning without attention, depth, or inference-time synaptic plasticity. Activation: dendritic in-context learning, DendriCL, single-layer SNN ICL, compartmental spiking neuron, apical LMS dynamics, Garg-2022 ICL benchmark, biologically plausible in-context learning.

Dendritic In-Context Learning in a Single-Layer Spiking Neural Network

DendriCL collapses general-purpose in-context learning (ICL) to a single compartmental spiking layer: the subthreshold dendritic/apical dynamics ARE the online learning algorithm (leaky Widrow-Hoff LMS), not a passive conduit for error signals. Beats dense Transformers on super-dimensional Garg-2022 ICL while being seed-stable and structurally interpretable.

Metadata

• Source: arXiv:2607.02283
• Authors: Juwei Shen, Yujie Wu, Changwen Chen
• Published: 2026-07-02
• Categories: cs.NE, cs.LG
• Benchmark: Garg-2022 in-context learning

Core Problem

In-context learning in modern AI (Transformers, Mamba, SSMs, MLPs) is widely believed to operate via implicit gradient descent embedded in the forward pass. Capturing ICL in biologically plausible Spiking Neural Networks (SNNs) was an open challenge:

1. Prior SNN-ICL attempts route adaptation through inference-time synaptic plasticity (STDP/Hebbian updates during test).
2. They treat the dendritic compartment as a passive conduit for error/teacher signals.
3. Result: they fail the Garg-2022 ICL benchmark at non-trivial task dimension — the linear regression / ridge tasks where ICL should shine.

Key Insight (the reframe)

The failure traces to a structural assumption, not a training issue:

"The subthreshold dynamics of a single dendritic compartment already implement a complete online learning algorithm."

Instead of viewing dendrites as passive wires carrying errors to synapses, treat the compartment itself as the computational substrate. A single-layer compartmental spiking neuron with apical recurrence has dynamics structurally identical to leaky online Widrow-Hoff LMS.

Core Methodology — DendriCL

Architecture

• Single-layer compartmental spiking network (no attention, no depth stack).
• Each neuron has a somatic spike-generating compartment + an apical/dendritic compartment whose subthreshold voltage integrates input.
• Apical recurrence: the apical state at time t is updated by a rule mathematically equal to: w(t+1) = w(t) + η · e(t) · x(t) (online LMS / Widrow-Hoff) where e(t) is the prediction error and x(t) the regressor — implemented purely by membrane/ionic dynamics, not by weight changes.
• No inference-time weight plasticity: weights are fixed; the "learning" lives in the dynamic state of the compartment.

Why this gives ICL

• In-context examples (x_i, y_i) are streamed as the input sequence.
• The apical compartment accumulates the online-LMS solution as a latent dynamical state.
• The query x_q is read out by a linear probe on the apical membrane.
• Because the update rule is embedded in the ODE, the network "solves" the in-context linear task during the forward pass — exactly the mechanistic definition of ICL.

Empirical results (from the paper)

• Seed-stable at super-dimensional Garg-2022 ICL where dense Transformers exhibit grokking-style instability and fail past moderate task dimension.
• A linear probe recovers the reference online-LMS trajectory directly from the apical membrane at R² = 0.93 — confirming the algorithm is structurally embedded in the dynamics, not implicitly discovered during training.
• Demonstrates ICL requires neither attention, nor depth, nor inference-time plasticity: a single compartment with online-LMS dynamics is sufficient.

How to Apply This Pattern

Use this when building or analyzing ICL-capable models, especially biologically plausible / neuromorphic ones:

1. Map the target algorithm to a dynamical system. Ask: "What ODE, if its state is read out linearly, reproduces the algorithm?" (Here: leaky LMS ↔ apical subthreshold dynamics.)
2. Promote the compartment from conduit to substrate. Don't pipe errors to synaptic updates; let the compartment's own evolution BE the update.
3. Keep weights frozen at inference; learn in latent state. This decouples "learning" from "plasticity" and removes the instability of inference-time weight changes.
4. Validate with a linear probe on the latent state to prove the algorithm is embedded (R² ≈ 1), not just approximated.
5. Benchmark on Garg-2022 (linear regression in-context) at increasing task dimension to expose where depth/attention-based ICL breaks (grokking) but the single-compartment model stays stable.

Pitfalls

• Treating dendrites as passive error carriers (the prior failed assumption) — the reframe is the whole contribution.
• Expecting weight changes during inference — DendriCL explicitly avoids them; "learning" = state evolution.
• Over-complicating the architecture: the win is minimalism (one layer, one compartment type). Adding depth defeats the point.

Verification

• Reproduce: stream Garg-2022 linear-regression prompts; confirm accuracy holds as task dimension grows past the Transformer failure point.
• Probe: fit a linear map from apical membrane voltage → online-LMS weight vector; expect R² ≳ 0.9.
• Ablate: remove apical recurrence → ICL collapses, confirming the dynamics (not the weights) carry the algorithm.

Related directions

• Compartmental spiking neurons (Hodgkin-Huxley multi-compartment) as differentiable learning substrates.
• Neuromorphic / energy-constrained ICL (pair with spiking-polar-trajectory-generator, parallel-tempering-snn-csp in this collection).
• Theory of ICL-as-dynamics vs ICL-as-implicit-gradient (connects to "contravariance theory" NeuroAI alignment work).