Brain-guided language model framework for robust reasoning - using task-fMRI signals from reasoning regions to enhance LLM performance across 10 models with up to 13% accuracy gain
Brain-guided language model framework for robust reasoning - using task-fMRI signals from reasoning regions to enhance LLM performance across 10 models with up to 13% accuracy gain
"""
Compute neural-predictivity score.
Args:
- model_representations: [batch, seq_len, hidden_dim]
- brain_activity: [batch, voxels, timepoints]
- aggregate: compute aggregate score vs. reasoning-type specific
- reasoning_specific: focus on reasoning regions
Returns:
- predictivity_score: alignment metric (0-1)
"""
# Encode model representations
self
# Encode brain activity (focus on specific regions)
if
self
else
self
self
# Compute voxel-wise encoding accuracy
self
# Aggregate or per-reasoning-type
if
return
else
# Break down by reasoning type (deductive, inductive, etc.)
return
self
def
compute_voxelwise_encoding
self, model_features, brain_features
"""
Compute encoding accuracy for each voxel.
Method: Train linear regression from model features to brain signals,
measure explained variance (R²).
"""
from
import
from
import
# Fit encoding model for each voxel
for
in
range
1
# Ridge regression
1.0
# Predict and measure R²
return
def
breakdown_by_type
self, predictivity, reasoning_types
"""
Analyze predictivity by reasoning category.
Args:
- reasoning_types: ['deductive', 'inductive', 'abductive', ...]
Returns:
- type_specific_scores: dict of predictivity per reasoning type
"""
for
in
# Extract voxels activated during specific reasoning type
self
return
2. Joint Structure Analysis
classJointStructureAnalysis:
"""
Analyze joint representation space of brain and model.
Key insight: Find directions where brain and model representations
align, then use these for steering interventions.
Method: Canonical Correlation Analysis (CCA) or Similarity Structure
Analysis on joint brain-model embedding space.
"""def__init__(
self,
brain_dim,
model_dim,
joint_dim=100):
self.joint_dim = joint_dim
# Projectors to joint spaceself.brain_projector = nn.Linear(brain_dim, joint_dim)
self.model_projector = nn.Linear(model_dim, joint_dim)
# CCA components (learned alignment directions)self.cca = Nonedefcompute_joint_structure(
self,
brain_representations,
model_representations
):
"""
Compute joint brain-model representation structure.
Args:
- brain_representations: [batch, voxels, time]
- model_representations: [batch, seq, hidden]
Returns:
- steering_directions: aligned directions in model space
- correlation_matrix: brain-model correlation structure
"""# Project to joint space
brain_joint = self.brain_projector(
brain_representations.mean(dim=-1) # average over time
)
model_joint = self.model_projector(
model_representations.mean(dim=1) # average over sequence
)
# Compute CCA to find aligned directionsself.cca = self.fit_cca(brain_joint, model_joint)
# Extract steering directions (highly correlated components)
steering_directions = self.extract_steering_directions(
self.cca,
threshold_correlation=0.5
)
return steering_directions
deffit_cca(self, X_brain, X_model):
"""
Fit Canonical Correlation Analysis.
"""from sklearn.cross_decomposition import CCA
cca = CCA(n_components=self.joint_dim)
cca.fit(X_brain, X_model)
return cca
defextract_steering_directions(self, cca, threshold=0.5):
"""
Extract model-space directions highly correlated with brain.
Returns directions in model hidden space that correspond to
reasoning-relevant brain patterns.
"""# Get canonical correlations
correlations = cca.score(X_brain, X_model)
# Select high-correlation components
high_corr_indices = np.where(correlations > threshold)[0]
# Extract corresponding directions in model space
steering_directions = cca.x_weights_[high_corr_indices]
return steering_directions
3. Steering at Inference
classBrainGuidedSteering:
"""
Intervene on LLM representations at inference time.
Method: Shift hidden representations along brain-induced directions
to enhance reasoning performance.
Key insight: Small shifts (scaled by brain predictivity) improve
reasoning without damaging language capabilities.
"""def__init__(
self,
steering_directions,
steering_scale=0.1,
intervention_layer=-1# apply to last layer or specific layer):
self.steering_directions = steering_directions
self.steering_scale = steering_scale
self.intervention_layer = intervention_layer
defapply_inference_steering(
self,
model,
input_ids,
reasoning_task_type
):
"""
Apply brain-guided steering during inference.
Args:
- model: LLM to enhance
- input_ids: input tokens
- reasoning_task_type: type of reasoning (deductive, etc.)
Returns:
- enhanced_output: steered model predictions
"""# Get base model representationswith torch.no_grad():
base_outputs = model(input_ids, output_hidden_states=True)
base_hidden = base_outputs.hidden_states[self.intervention_layer]
# Select task-specific steering direction
steering_vector = self.select_steering_vector(reasoning_task_type)
# Apply steering intervention
steered_hidden = base_hidden + self.steering_scale * steering_vector
# Continue inference from steered representations
enhanced_output = model.forward_from_hidden(
steered_hidden,
intervention_layer=self.intervention_layer
)
return enhanced_output
defselect_steering_vector(self, reasoning_task_type):
"""
Select steering direction specific to reasoning type.
Args:
- reasoning_task_type: 'deductive', 'inductive', etc.
Returns:
- steering_vector: direction in hidden space
"""# Map reasoning type to brain activation pattern
type_to_direction = {
'deductive': self.steering_directions['deductive'],
'inductive': self.steering_directions['inductive'],
'abductive': self.steering_directions['abductive'],
'default': self.steering_directions.mean(axis=0)
}
steering_vector = type_to_direction.get(
reasoning_task_type,
type_to_direction['default']
)
return steering_vector
defadaptive_scale(self, confidence_score):
"""
Adaptively scale steering based on model confidence.
Key insight: Apply stronger steering when model is uncertain.
"""# Inverse relationship: low confidence → strong steering
adaptive_scale = self.steering_scale * (1 - confidence_score)
return adaptive_scale
4. Brain-Signal Fine-Tuning
classBrainGuidedFineTuning:
"""
Fine-tune LLM using brain signals as additional supervision.
Method: Add neural-predictivity loss to standard language modeling loss.
Brain signals provide reasoning-specific guidance orthogonal to
language-only training.
"""def__init__(
self,
model,
neural_predictivity_metric,
reasoning_tasks,
alpha=0.5# brain guidance weight):
self.model = model
self.neural_predictivity = neural_predictivity_metric
self.reasoning_tasks = reasoning_tasks
self.alpha = alpha
deffine_tune(
self,
train_dataset,
brain_dataset,
num_epochs=10,
lr=5e-5):
"""
Fine-tune with brain signals.
Args:
- train_dataset: language data
- brain_dataset: task-fMRI data paired with reasoning tasks
Returns:
- fine_tuned_model: enhanced reasoning capability
"""
optimizer = torch.optim.AdamW(self.model.parameters(), lr=lr)
for epoch inrange(num_epochs):
for language_batch, brain_batch inzip(train_dataset, brain_dataset):
# Standard language modeling loss
lm_loss = self.compute_lm_loss(language_batch)
# Brain-guided reasoning loss
brain_loss = self.compute_brain_guided_loss(brain_batch)
# Combined loss
total_loss = lm_loss + self.alpha * brain_loss
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
defcompute_brain_guided_loss(self, brain_batch):
"""
Compute loss from neural-predictivity alignment.
Key insight: Encourage model representations to predict
reasoning-region brain activity.
"""# Get model representations
input_ids = brain_batch['input_ids']
outputs = self.model(input_ids, output_hidden_states=True)
hidden_states = outputs.hidden_states[-1]
# Get brain activity
brain_activity = brain_batch['fMRI']
reasoning_type = brain_batch['reasoning_type']
# Compute neural-predictivity
predictivity = self.neural_predictivity.compute_predictivity(
model_representations=hidden_states,
brain_activity=brain_activity,
aggregate=False,
reasoning_specific=True
)
# Loss: maximize predictivity in reasoning regions# Use negative predictivity as loss (minimize)
brain_loss = -predictivity.mean()
return brain_loss
Key Experimental Findings
Neural-Predictivity Analysis
Finding 1: LLMs explain substantial variance in reasoning regions
Aggregate level: High predictivity across reasoning cortex
Multimodal: Combine fMRI + EEG for enhanced guidance
Clinical: Apply to cognitive rehabilitation
References
Xiao et al. (2026). "Beyond Representational Alignment with Brain-Guided Language Models"
Neural encoding literature
Reasoning neuroscience meta-analyses
LLM steering methods
Citation
@article{xiao2026brainguided,
title={Beyond Representational Alignment with Brain-Guided Language Models for Robust Reasoning},
author={Xiao, Mingqing and Du, Kai and Lin, Zhouchen},
journal={arXiv preprint arXiv:2606.11893},
year={2026}
}
