cognition-inspired-dual-stream-emotion

name	cognition-inspired-dual-stream-emotion
description	Cognition-Inspired Dual-Stream Semantic Enhancement (DuSE) for Vision-Based Dynamic Emotion Modeling. Implements hierarchical temporal prompt clusters (HTPC) for cognitive priming and latent semantic emotion aggregators (LSEA) for knowledge integration. Models neuro-cognitive mechanisms from Conceptual Act Theory for dynamic facial expression recognition. Use for: emotion recognition, cognitive-inspired computer vision, neuro-cognitive modeling, dynamic facial expression analysis.
version	v1.0.0
last_updated	2026-04-15T00:00:00.000Z
paper	https://arxiv.org/abs/2604.12777v1

Cognition-Inspired Dual-Stream Emotion Modeling (DuSE)

Source Paper: Cognition-Inspired Dual-Stream Semantic Enhancement for Vision-Based Dynamic Emotion Modeling
Authors: Huanzhen Wang, Ziheng Zhou, Zeng Tao, Aoxing Li, Yingkai Zhao
Published: 2026-04-14

Overview

Models human brain emotion perception through dynamic, hierarchical integration of sensory input with semantic and contextual knowledge. Implements neuro-cognitive mechanisms from Conceptual Act Theory for dynamic facial expression recognition.

Biological Inspiration

The human brain constructs emotional percepts by:

1. Cognitive Priming: Linguistic cues pre-sensitize neural pathways (HTPC stream)
2. Knowledge Integration: Sensory inputs synthesized with learned concepts (LSEA stream)
3. Hierarchical Processing: Reflecting hippocampus and default mode network roles

Dual-Stream Architecture

Stream 1: Hierarchical Temporal Prompt Cluster (HTPC)

Purpose: Operationalize cognitive priming effect

Text Semantics ──► Fine-grained Temporal Features ──► Modulated Visual Processing

Mechanism:

• Linguistic cues pre-sensitize neural pathways
• Modulate processing of incoming visual stimuli
• Align textual semantics with temporal facial dynamics

Stream 2: Latent Semantic Emotion Aggregator (LSEA)

Purpose: Model knowledge integration (Conceptual Act Theory)

Sensory Inputs ──► Learned Conceptual Knowledge ──► Coherent Emotional Experience

Mechanism:

• Aggregates sensory inputs across modalities
• Synthesizes with learned conceptual knowledge
• Reflects hippocampus and default mode network function

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    Dual-Stream DuSE Model                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   ┌─────────────────┐                    ┌─────────────────┐   │
│   │   HTPC Stream   │                    │   LSEA Stream   │   │
│   │  (Cognitive     │                    │  (Knowledge     │   │
│   │   Priming)      │                    │   Integration)  │   │
│   │                 │                    │                 │   │
│   │ Text Semantics  │                    │ Sensory Inputs  │   │
│   │      │          │                    │       │         │   │
│   │      ▼          │                    │       ▼         │   │
│   │ Fine-grained    │                    │ Learned         │   │
│   │ Temporal Feats  │                    │ Conceptual      │   │
│   │      │          │                    │ Knowledge       │   │
│   │      ▼          │                    │       │         │   │
│   │ Modulate Visual │                    │       ▼         │   │
│   │ Processing      │                    │  Coherent       │   │
│   └────────┬────────┘                    │  Emotion Exp.   │   │
│            │                             └────────┬────────┘   │
│            │                                      │            │
│            └───────────────┬──────────────────────┘            │
│                            │                                   │
│                            ▼                                   │
│                    ┌───────────────┐                           │
│                    │  Fusion &     │                           │
│                    │  Decision     │                           │
│                    └───────────────┘                           │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Key Features

1. Neuro-Cognitive Plausibility

Explicitly models:

• Prefrontal-limbic interactions in emotion processing
• Hierarchical temporal dynamics across brain regions
• Conceptual Act Theory mechanisms

2. Semantic Context Integration

Unlike traditional approaches that process facial expressions in isolation:

• Integrates temporal dynamics with semantic knowledge
• Context-aware emotion modeling
• Cross-modal (text + vision) alignment

3. Improved Interpretability

Neuro-cognitive grounding provides:

• Clear mapping to known brain mechanisms
• Interpretable attention weights
• Modularity aligned with brain function

Workflow

Phase 1: HTPC Processing

class HTPC(nn.Module):
    """
    Hierarchical Temporal Prompt Cluster
    Models cognitive priming effect
    """
    def __init__(self):
        self.semantic_encoder = TextEncoder()
        self.temporal_aligner = TemporalAlignmentModule()
        
    def forward(self, text_input, temporal_features):
        # Encode text semantics
        semantic_repr = self.semantic_encoder(text_input)
        
        # Align with temporal features
        modulated_features = self.temporal_aligner(
            semantic_repr, temporal_features
        )
        
        return modulated_features

Phase 2: LSEA Processing

class LSEA(nn.Module):
    """
    Latent Semantic Emotion Aggregator
    Models knowledge integration
    """
    def __init__(self):
        self.sensory_encoder = VisualEncoder()
        self.conceptual_knowledge = ConceptMemory()
        
    def forward(self, visual_input):
        # Process sensory input
        sensory_repr = self.sensory_encoder(visual_input)
        
        # Integrate with conceptual knowledge
        conceptual_repr = self.conceptual_knowledge(sensory_repr)
        
        # Aggregate into coherent experience
        emotional_experience = self.aggregate(
            sensory_repr, conceptual_repr
        )
        
        return emotional_experience

Phase 3: Dual-Stream Fusion

class DuSE(nn.Module):
    """
    Dual-Stream Semantic Enhancement
    """
    def __init__(self):
        self.htpc = HTPC()
        self.lsea = LSEA()
        self.fusion = AttentionFusion()
        
    def forward(self, visual_input, text_input):
        # HTPC stream: cognitive priming
        htpc_output = self.htpc(text_input, visual_input)
        
        # LSEA stream: knowledge integration
        lsea_output = self.lsea(visual_input)
        
        # Fuse both streams
        output = self.fusion(htpc_output, lsea_output)
        
        return output

Results

• Benchmarks: State-of-the-art on challenging in-the-wild datasets
• Performance: Superior to vision-only approaches
• Interpretability: Enhanced through neuro-cognitive alignment

Activation Keywords

• dual stream emotion recognition
• cognition inspired vision model
• conceptual act theory implementation
• hierarchical temporal prompt cluster
• latent semantic emotion aggregator
• neuro cognitive emotion modeling
• dynamic facial expression recognition

References

• Paper: https://arxiv.org/abs/2604.12777v1
• PDF: https://arxiv.org/pdf/2604.12777v1
• arXiv ID: 2604.12777v1