pan-fm-pan-organ-foundation

name

pan-fm-pan-organ-foundation

description

Pan-Organ Foundation Model (Pan-FM) for multimodal biomedical imaging with missing-organ robustness. Pre-trained on seven organs (Brain, Heart, Adipose, Liver, Kidney, Spleen, Pancreas) using Saliency-Guided Masking (SGM) to prevent dominant-organ shortcut learning bias. Introduces masking-based self-distillation for handling realistic missing-organ scenarios. Use when: (1) building multi-organ medical imaging foundation models, (2) handling missing modalities/organs in biomedical data, (3) designing saliency-guided masking strategies, (4) developing whole-body representation learning systems, (5) addressing dominant-organ shortcut learning in multimodal pre-training. Activation keywords: pan-organ foundation model, missing organ robustness, saliency-guided masking, multimodal biomedical imaging, dominant-organ shortcut learning, whole-body representation learning, UK Biobank multi-organ.

Pan-FM: Pan-Organ Foundation Model with Saliency-Guided Masking

arXiv:2605.07055 | Wu, McIlvain, Yu, & Wen (2026)

Core Problem

Medical foundation models are trained on single organs in isolation, but human aging and disease arise from coordinated biological processes across organs. Real-world multimodal biomedical data have organs missing not at random (MNAR), causing:

• Reduced statistical power
• Limited generalizability
• Systematic bias

Key Innovation: Saliency-Guided Masking (SGM)

Dominant-Organ Shortcut Learning

Naive multimodal pre-training causes models to over-rely on dominant organs (adipose, heart) as shortcuts, ignoring subtler cross-organ signals.

SGM Mechanism

1. Compute attention distribution across organs during forward pass
2. Adaptively mask dominant organs based on attention scores
3. Force balanced learning — model must learn from all organs, not just the strongest signals
4. Negligible overhead — integrates seamlessly into existing self-supervised frameworks

# Pseudocode for Saliency-Guided Masking
def saliency_guided_masking(attention_weights, threshold=0.7, mask_prob=0.3):
    """Mask organs with disproportionately high attention."""
    # attention_weights: [batch, num_organs, seq_len]
    organ_attention = attention_weights.mean(dim=-1)  # [batch, num_organs]
    dominant_mask = organ_attention > threshold
    random_mask = torch.rand_like(organ_attention) < mask_prob
    # Only mask dominant organs probabilistically
    return dominant_mask & random_mask

Architecture

Component	Description
Unified backbone	Single architecture handling all 7 organs with organ-specific adapters
Masking-based self-distillation	Teacher-student framework with organ masking during pre-training
Missing-organ handling	Built into training AND inference — no post-hoc imputation needed
Cross-organ attention	Models inter-organ dependencies explicitly

Evaluation Results

• Dataset: UK Biobank
• Tasks: 13 disease categories + 14 single disease entities
• Baselines outperformed: Single-organ FMs and naive multi-organ models
• Key metric: Improved robustness under missing-organ settings

When to Use

Apply this skill when:

• Building foundation models spanning multiple organs/body systems
• Handling MNAR (Missing Not At Random) data in medical imaging
• Designing masking strategies for self-supervised learning
• Addressing modality imbalance in multimodal models
• Developing whole-body or system-level biomedical representations

Do NOT use when:

• Working with single-organ data only
• The task doesn't involve medical/biomedical imaging
• Missing data is MCAR (Missing Completely At Random) — simpler methods suffice

Implementation Patterns

Multi-Organ Pre-Training Pipeline

import torch

class PanFMPreTraining:
    def __init__(self, organs, backbone, sgm_threshold=0.7):
        self.organs = organs
        self.backbone = backbone
        self.sgm_threshold = sgm_threshold
    
    def sgm_mask(self, attention_scores):
        """Saliency-Guided Masking to prevent dominant-organ shortcuts."""
        organ_importance = attention_scores.mean(dim=-1)
        dominant = organ_importance > self.sgm_threshold
        mask = dominant & (torch.rand_like(organ_importance) < 0.3)
        return mask
    
    def forward_with_sgm(self, batch):
        # Standard forward pass
        features, attention = self.backbone(batch)
        # Apply SGM
        mask = self.sgm_mask(attention)
        # Masked self-distillation
        teacher_features = features.detach()
        student_features = self.backforward(batch, mask=mask)
        return self.distillation_loss(teacher_features, student_features)

Missing-Organ Inference

def predict_with_missing_organs(model, available_organs, missing_mask):
    """Handle inference when some organs are missing."""
    # Model handles missing organs natively — no imputation needed
    return model.predict(available_organs, mask=missing_mask)

Best Practices

1. Always use SGM when pre-training on heterogeneous multi-organ data
2. Monitor attention distributions — if one organ dominates >60% of attention, SGM is essential
3. Test missing-organ robustness as part of model evaluation
4. Report per-organ contribution to final predictions
5. Use UK Biobank as benchmark for multi-organ model comparison

Related Frameworks

• Single-organ FMs: brain-only, heart-only models (baseline comparison)
• Naive multi-organ: concatenation-based approaches (prone to shortcut learning)
• Self-supervised learning: DINO, MAE frameworks (SGM integrates with these)

arXiv Reference

• Paper: "Pan-FM: A Pan-Organ Foundation Model with Saliency-Guided Masking for Missing Robustness"
• Authors: Qiangqiang Wu, Grace McIlvain, Zhou Yu, Junhao Wen
• ID: arXiv:2605.07055v1 | Categories: cs.CV, cs.AI | Date: 2026-05-08
• Key contribution: First foundation model pre-trained on 7 organs with SGM for MNAR robustness

Related Skills

• tribe-v2-foundation-model: Multi-modal brain foundation model (video/audio/language)
• brain-foundation-biomarker-validation: Foundation model biomarker validation
• brain-foundation-model-batch-effects: Batch effects in brain foundation models