ai-interpretability-dead-salmon

name

ai-interpretability-dead-salmon

description

Statistical-causal reframing of AI interpretability: treating explanations as parameters of statistical models inferred from computational traces, with uncertainty quantification and testing against alternative computational hypotheses. Inspired by the famous 'dead salmon fMRI' study. Activation: dead salmon AI, interpretability statistics, causal interpretability, explanation uncertainty, statistical AI explanation, false discovery interpretability.

The Dead Salmons of AI Interpretability

Pragmatic statistical-causal reframing of AI interpretability: explanations should be treated as statistical model parameters inferred from computational traces, tested against alternative hypotheses with quantified uncertainty — guarding against "dead salmon" artifacts in interpretability research.

Metadata

• Source: arXiv:2512.18792
• Authors: Maxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard
• Published: 2025-12-21
• Category: cs.AI

Core Methodology

Key Insight

The famous "dead salmon fMRI" study (Bennett et al., 2009) showed that standard statistical analyses can produce seemingly meaningful brain activation patterns from a dead fish — a cautionary tale about misapplied statistical inference. The same artifact plagues AI interpretability: feature attribution, probing, sparse auto-encoding, and causal analyses can produce plausible-looking explanations for randomly initialized neural networks.

Statistical-Causal Reframing

1. Explanations as Statistical Parameters

   • Interpretability methods should be treated as statistical estimators
   • Explanations are parameters of a model, inferred from computational traces
   • Findings must be tested against explicit alternative computational hypotheses

2. Uncertainty Quantification

   • Go beyond measuring statistical variability due to finite input sampling
   • Quantify uncertainty with respect to the postulated statistical model
   • Report confidence intervals for attribution scores, not just point estimates

3. Identifiability Analysis

   • Many interpretability queries are fundamentally non-identifiable
   • Different computational hypotheses can produce identical explanations
   • Understanding identifiability limits is critical to avoiding false discoveries

4. Testing Framework

   • Establish null hypotheses (e.g., randomly initialized network)
   • Test explanations against these null models
   • Require explanations to exceed significance thresholds
   • Report effect sizes, not just statistical significance

Practical Guidelines

1. Always test against random baseline: Does your explanation method produce meaningful results on randomly initialized networks?
2. Report uncertainty: Include confidence intervals or credible intervals for attribution scores
3. Check identifiability: Can different models produce the same explanation? If so, the explanation is ambiguous
4. Use causal interventions: Verify explanations by intervening on the model (e.g., ablating features)
5. Multiple hypothesis correction: When testing many features/neurons, apply FDR correction

Code Example

# Pseudocode for statistically rigorous interpretability
import numpy as np

def dead_salmon_test(attribution_method, model, random_model, data, n_permutations=100):
    """Test if attribution method produces meaningful results beyond random baseline."""
    
    # Get attributions on real model
    real_attributions = attribution_method(model, data)
    
    # Get attributions on random baseline
    random_attributions = attribution_method(random_model, data)
    
    # Permutation test
    null_distribution = []
    for _ in range(n_permutations):
        shuffled_data = np.random.permutation(data)
        null_attr = attribution_method(model, shuffled_data)
        null_distribution.append(np.mean(np.abs(null_attr)))
    
    # Calculate p-value
    p_value = np.mean(np.abs(real_attributions) <= np.percentile(null_distribution, 95))
    
    return {
        "real_attribution_mean": np.mean(np.abs(real_attributions)),
        "random_attribution_mean": np.mean(np.abs(random_attributions)),
        "p_value": p_value,
        "significant": p_value < 0.05
    }

def confidence_interval_attribution(attribution_method, model, data, n_bootstrap=50):
    """Bootstrap confidence intervals for attribution scores."""
    bootstrap_scores = []
    for _ in range(n_bootstrap):
        resample = data[np.random.randint(0, len(data), len(data))]
        scores = attribution_method(model, resample)
        bootstrap_scores.append(scores)
    
    bootstrap_scores = np.array(bootstrap_scores)
    ci_lower = np.percentile(bootstrap_scores, , axis=)
    ci_upper = np.percentile(bootstrap_scores, , axis=)
    
     {: bootstrap_scores.mean(axis=), : ci_lower, : ci_upper}

Applications

• Feature attribution validation: Ensure SHAP, Integrated Gradients, etc. produce meaningful results
• Probing analysis: Verify that probing classifiers aren't finding spurious correlations
• Sparse auto-encoder interpretation: Validate that learned features correspond to real concepts
• Causal analysis: Test causal claims with proper statistical controls
• AI safety research: Rigorous evaluation of model behavior explanations

Pitfalls

• Computational cost: Permutation tests and bootstrapping are expensive
• Null model choice: The "random model" must be carefully defined
• Multiple comparisons: Testing many features requires correction
• Identifiability limits: Some questions fundamentally cannot be answered with current methods

Related Skills

• representation-steering
• transformer-prototype-readout
• self-verification