| name | brain-foundation-biomarker-validation |
| description | RE-CONFIRM framework for validating robustness of biomarkers discovered by brain foundation models from dynamic functional connectivity. Systematic evaluation of internal reliability, external reliability, and validity for clinical biomarkers. Activation: RE-CONFIRM, biomarker validation, brain foundation model, robust biomarkers, dynamic functional connectivity. |
RE-CONFIRM: Validating Robust Biomarkers from Brain Foundation Models
Systematic framework (RE-CONFIRM) for validating biomarkers discovered by brain foundation models from dynamic functional connectivity, evaluating internal reliability, external reliability, and validity across datasets and populations.
Metadata
- Source: arXiv:2604.22018v1
- Authors: Zijian Zeng, Yijie Dong, Zhenyu Liu, Jiaqi Wu, Xiaohan Cao, Xiao Xiang, Yijia Zhou, Xian Li, Shifu Chen, Qixiang Lin, Zexuan Zhu, Ziqiang Li, Zhongke Gao, Xiaowei Yu, Zhengqing Miao, Lianglong Sun, Hui Shen, Mingrui Xia, Yijun Zhang, Yong He, Jie Zhang
- Published: 2026-04-23
- Categories: q-bio.NC, cs.AI, cs.LG
Core Methodology
Problem Statement
Brain foundation models (FMs) show remarkable performance in predicting brain disorders from dynamic functional connectivity (FC), but discovered biomarkers often lack:
- Internal Reliability: Inconsistent across folds, subjects, or time
- External Reliability: Fail to generalize across datasets, sites, or scanners
- Validity: May not reflect true neurobiological phenomena
Existing validation approaches are ad-hoc, lacking systematic frameworks for FM-derived biomarkers.
Key Innovation
RE-CONFIRM framework provides six validation criteria:
- Intra-Subject Reliability: Consistency across multiple scans of same subject
- Cross-Fold Reliability: Consistency across train/test splits
- Cross-Site Reliability: Consistency across different imaging sites
- Cross-Dataset Reliability: Consistency across independent datasets
- Construct Validity: Correlation with known behavioral/clinical measures
- Biological Plausibility: Alignment with established neurobiological knowledge
Technical Framework
RE-CONFIRM Framework Architecture
┌─────────────────────────────────────────────────────────┐
│ RE-CONFIRM Framework │
├─────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Brain Foundation Model (FM) │ │
│ │ ┌─────────────┐ ┌───────────────────────┐ │ │
│ │ │ Dynamic │───→│ Latent Space │ │ │
│ │ │ FC Input │ │ (Biomarkers) │ │ │
│ │ └─────────────┘ └───────────────────────┘ │ │
│ └──────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ RE-CONFIRM Validation │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────┐ │ │
│ │ │ Internal Reliability (3 criteria) │ │ │
│ │ │ • Intra-Subject (ICC) │ │ │
│ │ │ • Cross-Fold (Stability) │ │ │
│ │ │ • Temporal (Test-Retest) │ │ │
│ │ └──────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────┐ │ │
│ │ │ External Reliability (2 criteria) │ │ │
│ │ │ • Cross-Site (Scanner effects) │ │ │
│ │ │ • Cross-Dataset (Generalization) │ │ │
│ │ └──────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌──────────────────────────────────────────┐ │ │
│ │ │ Validity (1 criterion) │ │ │
│ │ │ • Construct Validity (Behavioral corr) │ │ │
│ │ │ • Biological Plausibility │ │ │
│ │ └──────────────────────────────────────────┘ │ │
│ │ │ │
│ └──────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Validation Report │ │
│ │ ✓ PASS / ✗ FAIL per criterion │ │
│ │ Reliability Score: [0-1] │ │
│ │ Recommendation: Accept / Reject / Refine │ │
│ └──────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────┘
Validation Criteria Details
1. Intra-Subject Reliability (ICC)
ICC(2,1) = (BMS - WMS) / (BMS + (k-1)*WMS)
Where:
- BMS: Between-subject mean square
- WMS: Within-subject mean square
- k: Number of measurements per subject
Interpretation:
- ICC < 0.50: Poor
- 0.50 ≤ ICC < 0.75: Moderate
- 0.75 ≤ ICC < 0.90: Good
- ICC ≥ 0.90: Excellent
2. Cross-Fold Reliability (Stability)
Stability = 1 - ||w_train - w_test|| / ||w_train||
Where w are model weights or feature importance scores
3. Temporal Reliability (Test-Retest)
test_retest = corr(biomarker_session1, biomarker_session2)
Pearson or Spearman correlation across repeated measurements
4. Cross-Site Reliability
Harmonized_biomarker = Combat(biomarker)
Site_effect = 1 - var(harmonized) / var(raw)
Lower site effect = better generalization
5. Cross-Dataset Reliability
AUC_source = model.evaluate(dataset_source)
AUC_target = model.evaluate(dataset_target)
Generalization_gap = |AUC_source - AUC_target|
Gap < 0.05: Excellent
Gap < 0.10: Acceptable
Gap > 0.10: Poor
6. Construct Validity
correlation(biomarker, clinical_score)
correlation(biomarker, cognitive_performance)
Significant correlations support validity
Implementation Guide
Prerequisites
- Python >= 3.8
- PyTorch or TensorFlow for FM implementation
- Nilearn (for fMRI/FC processing)
- Pingouin or SciPy (for ICC)
- neuroCombat (for harmonization)
- scikit-learn (for metrics)
Step-by-Step Implementation
1. Data Preparation
import numpy as np
import pandas as pd
from nilearn.connectome import ConnectivityMeasure
class FCDataLoader:
"""
Load and preprocess dynamic functional connectivity data
"""
def __init__(self, atlas='schaefer', n_rois=400, window_length=30, step=15):
self.atlas = atlas
self.n_rois = n_rois
self.window_length = window_length
self.step = step
self.correlation_measure = ConnectivityMeasure(
kind='correlation',
vectorize=True
)
def load_time_series(self, bold_file, confounds_file=None):
"""
Load BOLD time series
Args:
bold_file: Path to preprocessed BOLD data
confounds_file: Path to confound regressors
Returns:
time_series: [n_volumes, n_rois]
"""
import nibabel as nib
img = nib.load(bold_file)
data = img.get_fdata()
time_series = self._extract_roi_timeseries(data)
if confounds_file:
confounds = pd.read_csv(confounds_file, sep='\t')
time_series = self._regress_confounds(time_series, confounds)
time_series
():
n_volumes = time_series.shape[]
n_windows = (n_volumes - .window_length) // .step +
dfc_windows = []
i (n_windows):
start = i * .step
end = start + .window_length
window_ts = time_series[start:end, :]
fc = np.corrcoef(window_ts.T)
fc = np.arctanh(np.clip(fc, -, ))
fc_vec = fc[np.triu_indices_from(fc, k=)]
dfc_windows.append(fc_vec)
np.array(dfc_windows)
():
dataset = {
: [],
: [],
: [],
: [],
: []
}
i, bold_file (file_list):
time_series = .load_time_series(bold_file)
dfc = .compute_dynamic_fc(time_series)
dataset[].append(dfc)
dataset[].append(labels[i] labels )
dataset[].append(sessions[i] sessions )
dataset[].append(sites[i] sites )
dataset[].append(i)
dataset
2. Foundation Model Implementation
import torch
import torch.nn as nn
class BrainFoundationModel(nn.Module):
"""
Brain foundation model for dynamic FC analysis
(Simplified example - actual FMs may be more complex)
"""
def __init__(self, input_dim, latent_dim=128, n_classes=2):
super().__init__()
self.temporal_encoder = nn.TransformerEncoder(
nn.TransformerEncoderLayer(d_model=input_dim, nhead=8, dim_feedforward=512),
num_layers=4
)
self.embedding = nn.Sequential(
nn.Linear(input_dim, 512),
nn.ReLU(),
nn.Linear(512, latent_dim)
)
self.biomarker_attention = nn.MultiheadAttention(latent_dim, num_heads=4)
self.classifier = nn.Sequential(
nn.Linear(latent_dim, 64),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(64, n_classes)
)
self.latent_dim = latent_dim
def forward(self, dfc, return_biomarkers=False):
"""
Args:
dfc: [batch, n_windows, n_features]
Returns:
logits: [batch, n_classes]
biomarkers: [batch, latent_dim] (optional)
"""
x = .temporal_encoder(dfc)
x = x.mean(dim=)
biomarkers = .embedding(x)
logits = .classifier(biomarkers)
return_biomarkers:
logits, biomarkers
logits
():
.()
biomarkers = []
labels = []
torch.no_grad():
batch dataloader:
dfc = batch[]
label = batch[]
_, bio = .forward(dfc, return_biomarkers=)
biomarkers.append(bio.cpu().numpy())
labels.append(label.cpu().numpy())
np.concatenate(biomarkers), np.concatenate(labels)
3. RE-CONFIRM Validation
import pingouin as pg
from scipy import stats
from neuroCombat import neuroCombat
from sklearn.metrics import roc_auc_score
class RECONFIRMValidator:
"""
RE-CONFIRM validation framework
"""
def __init__(self, model, threshold_icc=0.75, threshold_stability=0.8):
self.model = model
self.threshold_icc = threshold_icc
self.threshold_stability = threshold_stability
self.results = {}
def validate_intra_subject(self, biomarkers, subject_ids):
"""
Validate intra-subject reliability (ICC)
Args:
biomarkers: [n_samples, n_features]
subject_ids: [n_samples] subject identifiers
Returns:
icc_scores: [n_features] ICC for each biomarker dimension
"""
n_features = biomarkers.shape[1]
icc_scores = []
for i in range(n_features):
data = pd.DataFrame({
'subject': subject_ids,
'measurement': biomarkers[:, i]
})
try:
icc_result = pg.intraclass_corr(
data=data,
targets='subject',
raters='measurement',
ratings='measurement'
)
icc = icc_result['ICC'].iloc[2]
:
icc =
icc_scores.append(icc)
np.array(icc_scores)
():
sklearn.model_selection KFold
all_biomarkers = []
batch dataloader:
_, bio = .model(batch[], return_biomarkers=)
all_biomarkers.append(bio.detach().cpu().numpy())
all_biomarkers = np.concatenate(all_biomarkers)
n_samples = (all_biomarkers)
biomarker_folds = []
kf = KFold(n_splits=n_folds, shuffle=, random_state=)
train_idx, test_idx kf.split(all_biomarkers):
train_bio = all_biomarkers[train_idx]
train_mean = train_bio.mean(axis=)
test_bio = all_biomarkers[test_idx]
test_mean = test_bio.mean(axis=)
biomarker_folds.append((train_mean, test_mean))
stabilities = []
train_mean, test_mean biomarker_folds:
stability = - np.linalg.norm(train_mean - test_mean) / np.linalg.norm(train_mean)
stabilities.append(stability)
np.mean(stabilities), np.std(stabilities)
():
data_df = pd.DataFrame(biomarkers)
covars = pd.DataFrame({
: subject_ids,
: site_ids
})
harmonized = neuroCombat(
dat=data_df.T,
covars=covars,
batch_col=,
discrete_cols=,
continuous_cols=
)[].T
var_raw = np.var(biomarkers, axis=)
var_harmonized = np.var(harmonized, axis=)
site_effect = - (var_harmonized / (var_raw + ))
harmonized, site_effect
():
bio_source, labels_source = model.extract_biomarkers(dataset_source)
sklearn.linear_model LogisticRegression
clf = LogisticRegression()
clf.fit(bio_source, labels_source)
auc_source = roc_auc_score(labels_source, clf.predict_proba(bio_source)[:, ])
bio_target, labels_target = model.extract_biomarkers(dataset_target)
auc_target = roc_auc_score(labels_target, clf.predict_proba(bio_target)[:, ])
generalization_gap = (auc_source - auc_target)
{
: auc_source,
: auc_target,
: generalization_gap
}
():
correlations = []
p_values = []
i (biomarkers.shape[]):
corr, pval = stats.pearsonr(biomarkers[:, i], clinical_scores)
correlations.append(corr)
p_values.append(pval)
np.array(correlations), np.array(p_values)
():
biomarkers = dataset[]
subject_ids = dataset[]
sites = dataset.get()
report = {
: {},
: ,
:
}
dataset:
icc = .validate_intra_subject(biomarkers, dataset[])
report[][] = {
: np.mean(icc),
: np.median(icc),
: np.(icc >= .threshold_icc),
: np.median(icc) >= .threshold_icc
}
sites (np.unique(sites)) > :
_, site_effect = .validate_cross_site(biomarkers, sites, subject_ids)
report[][] = {
: np.mean(site_effect),
: np.mean(site_effect) <
}
clinical_scores :
corr, pval = .validate_construct_validity(biomarkers, clinical_scores)
significant = np.(pval < )
report[][] = {
: significant,
: np.mean(np.(corr)),
: significant > (corr) *
}
scores = [c.get() == c report[].values()]
report[] = np.mean(scores) scores
report[] >= :
report[] =
report[] >= :
report[] =
:
report[] =
.results = report
report
():
(output_file, ) f:
f.write(* + )
f.write()
f.write(* + )
f.write()
f.write()
f.write(* + )
f.write()
f.write(* + )
criterion, details .results[].items():
f.write()
key, value details.items():
(value, ):
f.write()
:
f.write()
Applications
- Clinical Biomarker Discovery: Validate brain disorder biomarkers
- Multi-Site Studies: Ensure reliability across imaging centers
- Longitudinal Tracking: Monitor biomarker stability over time
- Model Selection: Compare brain foundation models
- Regulatory Compliance: Meet validation standards for clinical use
Key Features
- Six Validation Criteria: Comprehensive reliability and validity assessment
- Quantitative Scoring: Objective pass/fail thresholds
- Harmonization Support: Built-in ComBat for site effects
- Modular Design: Can validate any brain FM
- Report Generation: Automated validation reports
Pitfalls
- Data Requirements: Needs test-retest data for intra-subject reliability
- Multi-Site Data: Cross-site validation requires multiple sites
- Clinical Scores: Construct validity needs behavioral/clinical data
- Computational Cost: Running all criteria on large datasets is expensive
- Threshold Sensitivity: Results depend on chosen ICC/stability thresholds
Related Skills
- brain-dit-fmri-foundation-model
- brain-mri-foundation-clinical
- functional-connectome-fingerprint
References
Zeng, Z., et al. (2026). RE-CONFIRM: Validating Robust Biomarkers Discovered
by Brain Foundation Models from Dynamic Functional Connectivity.
arXiv preprint arXiv:2604.22018v1.
