momenta-multimodal-moe-misinformation-detection

# MOMENTA: Multimodal MoE Misinformation Detection ## Overview Unified multimodal misinformation detection framework combining: - Modality-specific MoE modules for specialized processing - Bidirectional co-attention for text-visual alignment - Discrepancy-aware branch for cross-modal disagreement - Attention-based temporal aggregation with drift/momentum encoding - Domain-adversarial learning with prototype memory bank ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ Input Modalities │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Text │ │ Visual │ │ Temporal │ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ ↓ ↓ ↓ │ │ ┌──────────┐ ┌──────────┐ │ │ │ Text MoE │ │Vis MoE │ ← Modality-specific experts │ │ └────┬─────┘ └────┬─────┘ │ │ └──────┬──────┘ │ │ ↓ │ │ ┌─────────────────────┐ │ │ │ Bidirectional │ ← Cross-modal alignment │ │ │ Co-Attention │ │ │ └──────────┬──────────┘ │ │ ↓ │ │ ┌─────────────────────┐ ← Detect contradictions │ │ │ Discrepancy-Aware │ between modalities │ │ │ Branch │ │ │ └──────────┬──────────┘ │ │ ↓ │ │ ┌─────────────────────┐ ← Drift/momentum encoding │ │ │ Temporal Aggregation│ for temporal sequences │ │ │ (Drift/Momentum) │ │ │ └──────────┬──────────┘ │ │ ↓ │ │ ┌─────────────────────┐ ← Domain generalization │ │ │ Domain-Adversarial │ + prototype memory │ │ │ + Prototype Memory │ │ │ └──────────┬──────────┘ │ │ ↓ │ │ Misclassification (Real/Fake) │ └─────────────────────────────────────────────────────────┘ ``` ## Key Components ### 1. Modality-Specific MoE - Separate expert pools for text and visual modalities - Dynamic routing selects relevant experts per input - Enables specialized feature extraction ### 2. Bidirectional Co-Attention - Text attends to visual features AND visual attends to text - Creates aligned cross-modal representations - Captures inter-modal dependencies ### 3. Discrepancy-Aware Branch - Specifically detects disagreements between modalities - Key signal: text says one thing, image shows another - Primary misinformation indicator ### 4. Temporal Aggregation - Drift encoding: captures distribution shift over time - Momentum encoding: maintains temporal consistency - Attention-weighted aggregation of temporal features ## Evaluated Datasets - Fakeddit, MMCoVaR, Weibo, XFacta ## Applications - Social media content moderation - Automated fact-checking systems - News verification platforms - Multimodal content analysis ## Activation Keywords - misinformation detection, multimodal MoE, fact-checking - cross-modal disagreement, temporal aggregation - fake news detection, MOMENTA - 虚假信息检测, 多模态专家混合 ## References - Yeganeh Abdollahinejad, Ahmad Mousavi, et al. "MOMENTA: Mixture-of-Experts Over Multimodal Embeddings with Neural Temporal Aggregation for Misinformation Detection." arXiv:2604.16172