| name | neurrator-single-cell-semantic-narration |
| description | NEURRATOR - Semantic narration of vision at single-cell resolution. Maps spiking activity to natural-language descriptions via CLIP-LLaVA embedding space, enabling functional probing of cell types and brain regions. |
| version | 1.0.0 |
| category | neuroscience |
| arxiv_id | 2606.18667 |
| authors | Arnau Marin-Llobet, Richard Hakim, Sara Matias, Venkatesh N. Murthy, Na Li, Demba Ba |
| institution | Harvard University (Kempner Institute, Center for Brain Science) |
| published | 2026-06-17T00:00:00.000Z |
| activation_words | ["neurrator","semantic narration","single-cell decoding","neuron language","CLIP neural decoding","spike-to-text","cell-type functional probe","visual cortex narration","Neuropixel decoding","neural semantic"] |
| related_skills | ["spike-image-decoder","brain-cause-causal-visual-representation","neural-code-speak","neurrate-single-cell-semantic-narration","llm-eeg-graph-refinement"] |
NEURRATOR: Semantic Narration of Vision at Single-Cell Resolution
Core Innovation
NEURRATOR 是首个将单神经元脉冲活动直接映射到自然语言叙述的框架，实现从视觉皮层神经元活动到语义描述的端到端解码。
突破性贡献:
- 语义级解码器: 从单细胞脉冲活动生成连贯的自然语言描述
- 区域/细胞类型作为功能探针: 输入端使用细胞身份，输出端返回功能描述
- 概念级分解: 结合稀疏自编码器(SAE)将细胞贡献分解为可解释的视觉概念特征
Methodology
Architecture
Spike Trains → [Neurrator Encoder] → CLIP Patch Embeddings → [Frozen LLaVA] → Natural Language
核心组件:
-
Neurrator Encoder:
- Multi-scale 1D-CNN spike-train frontend
- Transformer temporal encoder
- Attention-weighted temporal pooling
- 576 learned patch queries (24×24 grid, 1024-d)
-
PatchInjector Hook: Runtime bypass of LLaVA vision tower
-
Frozen LLaVA-1.5-7B: Multimodal projector + LLaMA-2-7B decoder (no training)
Training Objective
L = 0.5 * MSE(P̂_t, P_f(t)) + 0.5 * (1 - cos(P̂_t, P_f(t)))
Dual loss: MSE + cosine similarity on predicted vs. true CLIP patch tensors.
Natural Language Decoding
- Greedy decoding with fixed one-sentence prompt
- No language-side training
- Patch tensor is the only modality bridge
Key Results
Decoding Fidelity Scaling
| Population | Semantic Accuracy | Generalization |
|---|
| 1000s neurons | High | Across held-out frames |
| Single region | Moderate | Across movies |
| Single neuron | Low but interpretable | N/A |
Region-Level Findings
- Higher visual cortex: Better semantic fidelity
- Primary visual cortex: Lower but still functional
- Cell-type signatures: Distinct concept profiles
Concept-Level Decomposition
Using CLIP SAE:
