automated-neural-characterization-language

name

automated-neural-characterization-language

description

Automated neural characterization using natural language and digital twins. Closed-loop framework that translates neuron activation patterns into concise semantic descriptions, generates hypothesis images, and verifies them in silico. Use when studying: neural selectivity characterization, digital twin neuroscience, semantic hypothesis testing, V1/V4 visual cortex encoding, generative models for neural decoding, or combining language models with neural data. arXiv: 2605.12485 (q-bio.NC, q-bio.QM). Lad, Franke, Rott Shaham, Ganguli, Tolias, Sanborn, Karantzas.

Automated Neural Characterization via Language

Closed-loop framework that uses natural language to characterize individual neuron selectivity at scale, combining generative models with neural digital twins.

Source: arXiv 2605.12485v1 (2026-05-12), q-bio.NC, q-bio.QM

Core Problem

Traditional receptive field models (e.g., Gabor functions for V1) fail in higher visual areas. No comparable analytical framework exists for areas like V4 where neurons respond to complex conjunctions of form, color, and texture.

Framework Architecture

The method operates as a closed loop with three stages:

1. Semantic Translation: Each neuron's high-activating and low-activating images are fed to a vision-language model that generates dense captions and semantic hypotheses about what the neuron responds to.

2. Hypothesis Synthesis: Generated descriptions drive a text-to-image model to produce synthetic test images that maximize (activating) or minimize (suppressing) the hypothesized features.

3. In Silico Verification: Synthetic images are presented to a digital twin of the neuron (a pretrained encoding model) to verify whether the hypothesis holds.

Key Results

• V4: 96.1% of neurons driven above 95th percentile by activating hypothesis images; 97.6% driven below 5th percentile by suppressing hypotheses (vs ~10% for random images).
• V1: Activation results matched V4 levels; suppression was less describable in language, consistent with V1 encoding more sub-linguistic features (oriented edges, spatial frequency).
• Representational alignment: RSA showed partial alignment between neural activity, vision embeddings, and language embeddings. Vision embeddings were most aligned with neural activity. Alignment lost in the text bottleneck was recovered when hypotheses were rendered back into images — linguistic compression is lossy yet semantically faithful.

Workflow for Agents

When to apply this framework

1. Characterizing neuron selectivity in higher visual areas
2. Building interpretable models of neural function
3. Scaling from hand-tuned models (Gabor) to automated description
4. Combining LLM/VLM reasoning with neural encoding models

Required components

neural_digital_twin  → pretrained encoding model (predicts neuron response to any image)
vision_language_model → generates semantic descriptions from image sets
text_to_image_model   → synthesizes images from semantic hypotheses

Step-by-step procedure

1. Collect high-activating and low-activating natural images for each neuron
2. Generate dense captions for each image set using a VLM
3. Synthesize a semantic hypothesis (concise description of selectivity)
4. Generate test images from activating and suppressing hypotheses
5. Present test images to the digital twin and measure response statistics
6. Verify: % of neurons exceeding natural-image response percentiles
7. Perform RSA between neural responses, vision embeddings, and language embeddings

Description taxonomy

Area	Typical descriptions
V1	Oriented edges, spatial frequency, bar orientation
V4	Conjunctions of form + color + texture, complex shapes

Activation Keywords

• neural characterization, neural selectivity, digital twin neuroscience
• semantic hypothesis testing, language-based neural decoding
• V1 V4 visual cortex encoding, neuron description
• automated neural analysis, closed-loop neural characterization

Tools Used

• vision-language models (VLM) for image captioning and hypothesis generation
• Text-to-image models for synthetic stimulus generation
• Neural encoding models (digital twins) for in silico verification
• Representational Similarity Analysis (RSA) for cross-modal alignment

Notes

• The framework is generalizable beyond visual cortex — applicable to any sensory area where a digital twin encoding model exists
• Linguistic compression is lossy: some neural information is lost in the text bottleneck, but recovered when hypotheses are rendered back into visual form
• Enables agentic scientific discovery at the neuron-characterization level
• Digital twin quality is the primary bottleneck for generalization