メニュー
Whisper-ECoG alignment methodology mapping speech foundation model representations to human cortical activity using interpretable time-resolved neural encoding
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/hiyenwong/ai_collection --skill whisper-ecog-alignmentコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SKILL.md を表示中
SOC 職業分類に基づく
| name | whisper-ecog-alignment |
| description | Whisper-ECoG alignment methodology mapping speech foundation model representations to human cortical activity using interpretable time-resolved neural encoding |
| category | neuroscience |
| activation_keywords | ["whisper","ecog","speech encoding","brain alignment","speech foundation model","temporal encoding","phoneme organization","cortical speech processing","time-resolved neural encoding","soft attention","hierarchical brain alignment"] |
| version | 1.0.0 |
| paper_id | arXiv:2606.02305 |
| authors | ["Matteo Ciferri","Tommaso Boccato","Michal Olak","Matteo Ferrante","Nicola Toschi"] |
| date | 2026-06-01T00:00:00.000Z |
| conference | ICLR 2026 Workshop on Representational Alignment (Re-Align) |
Mapping Whisper Representations to Human ECoG Responses - Speech foundation models (Whisper) provide a useful framework for studying time-resolved cortical speech representations, with intermediate layers showing strongest brain alignment.
Whisper Layer Correspondence with Neural Activity:
# Layer-wise encoding performance
layer_alignment = {
'early_layers': 0.45, # Low-level acoustic features
'intermediate_layers': 0.72, # Peak alignment
'late_layers': 0.58 # Semantic/linguistic features
}
Novel Encoder Components:
class TimeResolvedNeuralEncoder(nn.Module):
"""
Interpretable neural encoder combining:
1. Speech embeddings (from Whisper)
2. Recurrent temporal model (for dynamics)
3. Soft attention (for temporal alignment)
"""
def __init__(self, whisper_dim, hidden_dim, attention_heads):
super().__init__()
# Speech embedding processor
self.embedding_layer = nn.Linear(whisper_dim, hidden_dim)
# Temporal dynamics model
self.recurrent_model = nn.LSTM(hidden_dim, hidden_dim, batch_first=True)
.attention = nn.MultiheadAttention(hidden_dim, attention_heads)
.neural_decoder = nn.Linear(hidden_dim, num_ecog_channels)
():
processed = .embedding_layer(whisper_embeddings)
temporal_features, _ = .recurrent_model(processed)
attended, attention_weights = .attention(
temporal_features, temporal_features, temporal_features
)
ecog_predictions = .neural_decoder(attended)
ecog_predictions, attention_weights
Why This Architecture Works:
Anatomically Coherent Phoneme Organization:
# Encoding-informative electrodes show phoneme category organization
phoneme_categories = {
'obstruents': ['frontal_left_channels'], # Anatomically grouped
'sonorants': ['temporal_right_channels'],
'vowels': ['auditory_cortex'],
'consonants': ['motor_areas']
}
# Interpretability diagnostic
def analyze_phoneme_organization(electrodes, phoneme_labels):
"""
Identify anatomically coherent phoneme-category organization
among encoding-informative electrodes
"""
# Compute encoding scores per electrode
encoding_scores = compute_encoding_performance(electrodes)
# Select encoding-informative electrodes (top performers)
informative_electrodes = select_top_k(encoding_scores, k=20)
# Cluster by phoneme preference
phoneme_clusters = cluster_by_phoneme_response(
informative_electrodes, phoneme_labels
)
# Verify anatomical coherence
anatomical_coherence = verify_anatomical_grouping(phoneme_clusters)
return phoneme_clusters, anatomical_coherence
Interpretability Result: Electrodes that best encode speech show organized phoneme categories aligned with anatomical regions.
import whisper
def extract_whisper_embeddings(audio_path, model_size='base'):
"""
Extract layer-wise Whisper embeddings for speech segments
Returns embeddings from all transformer layers
"""
model = whisper.load_model(model_size)
audio = whisper.load_audio(audio_path)
# Encode and extract intermediate representations
embeddings = {}
for layer_idx in range(model.dims.n_layers):
embeddings[f'layer_{layer_idx}'] = model.encoder(
audio, layer_output=layer_idx
)
return embeddings
def process_ecog_data(raw_ecog, sampling_rate):
"""
Preprocess intracranial ECoG recordings
Key steps:
1. Bandpass filter (speech-relevant frequencies)
2. Normalize across channels
3. Segment by speech timeline
"""
# Filter to speech-relevant bands
filtered = bandpass_filter(raw_ecog, lowcut=1, highcut=100, fs=sampling_rate)
# Normalize
normalized = z_score_normalize(filtered, axis=1)
# Align with speech timeline
aligned_ecog = align_with_audio_segments(normalized, audio_timeline)
return aligned_ecog
def train_encoder(whisper_embeddings, ecog_data, epochs=100):
"""
Train time-resolved neural encoder
Loss: Predict actual ECoG responses from Whisper embeddings
"""
encoder = TimeResolvedNeuralEncoder(
whisper_dim=512, hidden_dim=256, attention_heads=4
)
optimizer = torch.optim.Adam(encoder.parameters(), lr=1e-4)
for epoch in range(epochs):
predictions, attention_weights = encoder(
whisper_embeddings, ecog_data.timestamps
)
loss = prediction_loss(predictions, ecog_data.signals)
optimizer.zero_grad()
loss.backward()
optimizer.step()
return encoder, attention_weights
def analyze_layer_alignment(encoder, whisper_embeddings, ecog_data):
"""
Compare encoding performance across Whisper layers
Find intermediate layers with strongest brain alignment
"""
layer_scores = {}
for layer_name, embeddings in whisper_embeddings.items():
predictions, _ = encoder(embeddings, ecog_data.timestamps)
score = compute_correlation(predictions, ecog_data.signals)
layer_scores[layer_name] = score
# Identify peak alignment
best_layer = max(layer_scores, key=layer_scores.get)
return layer_scores, best_layer
def phoneme_interpretability(encoder, electrodes, phoneme_labels):
"""
Analyze phoneme-category organization in encoding electrodes
"""
# Electrode-wise encoding performance
electrode_scores = compute_per_electrode_encoding(encoder)
# High-performing electrodes
informative_electrodes = electrodes[electrode_scores > threshold]
# Phoneme response patterns per electrode
phoneme_responses = extract_phoneme_responses(informative_electrodes)
# Cluster by phoneme preference
clusters = cluster_electrodes_by_phoneme(phoneme_responses)
# Verify anatomical coherence
coherence = check_anatomical_grouping(clusters)
return clusters, coherence
Explanation:
# Conceptual model of cortical hierarchy
cortical_hierarchy = {
'primary_auditory': {'level': 0, 'features': 'acoustic'},
'secondary_auditory': {'level': 1, 'features': 'phoneme'},
'association_areas': {'level': 2, 'features': 'semantic'},
}
# Whisper hierarchy (approximate)
whisper_hierarchy = {
'layers_0-6': {'level': 0, 'features': 'acoustic'},
'layers_7-12': {'level': 1, 'features': 'intermediate'}, # Peak alignment
'layers_13-24': {'level': 2, 'features': 'semantic'},
}
Key Finding: High-resolution ECoG benefits from temporally structured modeling beyond linear mappings.
# Comparison results
encoding_methods = {
'linear_mapping': {'score': 0.52},
'temporal_encoder': {'score': 0.72}, # +20% improvement
'linear_with_attention': {'score': 0.65}
}
Why: ECoG captures temporal dynamics at millisecond resolution → simple linear mappings miss temporal structure.
Attention Map Interpretation:
Use Case: Evaluate Whisper (and other speech models) for biological plausibility.
# Brain alignment as evaluation metric
def evaluate_speech_model(model, ecog_dataset):
"""
Assess biological plausibility via brain alignment
"""
encoder = TimeResolvedNeuralEncoder(model.embedding_dim)
alignment_score = train_and_score(encoder, model, ecog_dataset)
return alignment_score
Use Case: Decode neural activity to predict perceived speech.
# Inverse: From ECoG to speech
class NeuralToSpeechDecoder(nn.Module):
def __init__(self):
self.neural_encoder = TimeResolvedNeuralEncoder(...)
self.speech_decoder = WhisperDecoder()
def decode(self, ecog_signals):
neural_features = self.neural_encoder.inverse(ecog_signals)
predicted_speech = self.speech_decoder(neural_features)
return predicted_speech
Use Case: Map phoneme representations to cortical regions.
# Phoneme-to-region mapping
phoneme_cortex_map = {
'obstruents': 'left_frontal',
'vowels': 'auditory_cortex',
'nasals': 'motor_areas'
}
Problem: Training encoder on single "best" layer without exploring alternatives.
Solution:
# Layer exploration strategy
best_layers = []
for layer in all_whisper_layers:
score = evaluate_encoder_on_layer(layer)
if score > threshold:
best_layers.append(layer)
# Use multiple top layers for robustness
multi_layer_encoder = combine_top_layers(best_layers[:3])
Problem: Using only linear mappings without temporal modeling.
Solution: Always include recurrent temporal component for ECoG:
# Minimal temporal component
minimal_encoder = nn.Sequential(
nn.Linear(embedding_dim, hidden_dim),
nn.LSTM(hidden_dim, hidden_dim), # Essential for ECoG
nn.Linear(hidden_dim, ecog_channels)
)
Problem: Treating attention peaks as "true" alignment without validation.
Solution: Validate attention against ground-truth speech timeline:
def validate_attention(attention_weights, speech_timeline):
"""
Check if attention peaks align with actual speech events
"""
peaks = extract_attention_peaks(attention_weights)
speech_events = identify_speech_events(speech_timeline)
alignment = compute_peak_event_alignment(peaks, speech_events)
assert alignment > 0.7, "Attention validation failed"
Problem: Finding phoneme clusters that don't reflect actual phoneme processing.
Solution: Cross-validate with behavioral phoneme tasks:
def validate_phoneme_clusters(clusters, behavioral_data):
"""
Verify phoneme organization reflects actual phoneme perception
"""
# Compare with phoneme discrimination performance
discrimination_scores = behavioral_data.phoneme_discrimination
correlation = correlate_clusters_with_behavior(clusters, discrimination_scores)
return correlation > 0.6 # Threshold for validity
encoding_quality_thresholds = {
'poor': 0.4,
'moderate': 0.55,
'good': 0.65,
'excellent': 0.72 # As reported in paper
}
# ECoG requires millisecond-level temporal precision
def check_temporal_precision(encoder, ecog_data):
"""
Verify encoder captures millisecond-level dynamics
"""
predictions = encoder.predict(ecog_data)
# Compare temporal structure
temporal_correlation = compute_temporal_structure_correlation(
predictions, ecog_data
)
return temporal_correlation > 0.8
# Anatomical coherence check
def verify_anatomical_coherence(phoneme_clusters):
"""
Ensure phoneme categories align with known cortical regions
"""
for category, electrodes in phoneme_clusters.items():
anatomical_region = get_anatomical_region(electrodes)
# Check consistency
assert electrodes_same_region(anatomical_region)
Primary: whisper-ecog-alignment, speech encoding, brain alignment, temporal encoder
Secondary: speech foundation model, ecog, phoneme organization, cortical speech, soft attention, hierarchical alignment
Whisper-ECoG Alignment demonstrates that speech foundation models (Whisper) offer a useful framework for studying cortical speech representations. Key findings:
Impact: Speech foundation models bridge computational neuroscience and AI, enabling interpretable neural encoding research.