hormone-t5-emotion-layer

name

hormone-t5-emotion-layer

description

Hormone-inspired Emotion Layer for Transformer language models (HELT / HormoneT5). Biologically-inspired architecture augmenting transformers with a Hormone Emotion Block simulating the endocrine system's role in emotional processing. Six continuous hormone-like values computed via specialized per-hormone attention heads with orthogonally initialized learnable queries, temperature-scaled attention, and deep output projections. Emotional embedding modulates encoder hidden states for emotionally-appropriate response generation. Multi-objective training: sequence-to-sequence loss + hormone prediction loss with margin penalties + diversity regularization. Achieves 85%+ per-hormone accuracy. Use when: designing emotionally intelligent LLMs, affective computing, hormone-inspired neural architectures, bio-inspired emotion modeling in AI, transformer emotion augmentation. Keywords: HormoneT5, HELT, emotion layer, hormone attention, affective computing, endocrine-inspired AI, emotional language models, bio-inspired emotion.

HormoneT5: Hormone-inspired Emotion Layer for Transformers

arXiv: 2605.13858 | Reda & El-Metwally (April 2026)

Core Contribution

Augments transformer language models with a Hormone Emotion Block that simulates the human endocrine system's role in emotional processing, enabling continuous multi-dimensional emotion representation rather than discrete emotion classification.

Architecture

Hormone Emotion Block (HEB)

Input: Encoder hidden states (seq_len, d_model)
        ↓
┌─────────────────────────────────────────────┐
│         Hormone Emotion Block               │
│                                             │
│  6 Per-Hormone Attention Heads:            │
│  - Orthogonally initialized learnable queries│
│  - Temperature-scaled attention             │
│  - Deep output projections                  │
│                                             │
│  Output: 6 continuous hormone values        │
└─────────────────────────────────────────────┘
        ↓
Hormone values → Emotional embedding (d_model)
        ↓
Modulate encoder hidden states via addition/concatenation
        ↓
Emotionally-appropriate response generation

Six Hormone Dimensions

The six hormone-like values simulate endocrine dynamics:

1. Dopamine-like: Reward/pleasure signaling
2. Serotonin-like: Mood stabilization
3. Oxytocin-like: Social bonding/empathy
4. Cortisol-like: Stress response
5. Adrenaline-like: Arousal/excitement
6. Endorphin-like: Pain relief/comfort

Per-Hormone Attention Heads

Each hormone dimension uses a dedicated attention head with:

• Orthogonally initialized learnable queries: Ensures hormone diversity from the start
• Temperature-scaled attention: Controls hormone specificity vs. generality
• Deep output projections: Maps attention patterns to continuous hormone values

Training Framework

Multi-Objective Loss

total_loss = (
    seq2seq_loss          # Standard language modeling loss
    + lambda_h * hormone_loss    # Hormone prediction with margin penalties
    + lambda_d * diversity_loss  # Prevents attention collapse
)

1. Sequence-to-sequence loss: Standard cross-entropy for text generation
2. Hormone prediction loss: Margin-based loss ensuring correct hormone prediction within tolerance (0.15 threshold)
3. Diversity regularization: Prevents hormone attention heads from collapsing to similar patterns

Training Results

• 85%+ per-hormone accuracy within 0.15 tolerance
• Hormone differentiation range > 0.85 across all six hormones between contrasting tones
• Human evaluation: Significant preference (p < 0.01) for emotional appropriateness and empathetic quality vs. baseline T5

Implementation Patterns

Hormone Block Integration

class HormoneEmotionBlock(nn.Module):
    """Hormone-inspired emotion modulation for transformers."""
    
    def __init__(self, d_model: int, n_hormones: int = 6):
        super().__init__()
        self.n_hormones = n_hormones
        # Per-hormone attention with orthogonal initialization
        self.hormone_queries = nn.Parameter(
            torch.nn.init.orthogonal_(torch.empty(n_hormones, d_model))
        )
        self.temperature = nn.Parameter(torch.ones(n_hormones))
        self.projection = nn.Sequential(
            nn.Linear(d_model, d_model * 2),
            nn.GELU(),
            nn.Linear(d_model * 2, 1)  # scalar hormone value
        )
        self.emotion_proj = nn.Linear(n_hormones, d_model)
    
    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # Compute per-hormone attention
        # temperature-scaled, with deep projection
        hormone_values = self._compute_hormones(hidden_states)
        # Convert to emotional embedding
        emotion_embedding = self.emotion_proj(hormone_values)
        return emotion_embedding

Multi-Objective Training

def hormone_loss(predicted, target, margin=0.15):
    """Margin-based hormone prediction loss."""
    diff = torch.abs(predicted - target)
    return torch.mean(torch.relu(diff - margin))

def diversity_loss(hormone_attentions):
    """Prevents attention collapse across hormone heads."""
    # Cosine similarity between hormone query projections
    normalized = F.normalize(hormone_attentions, dim=-1)
    similarity = torch.matmul(normalized, normalized.transpose(-1, -2))
    # Penalize high similarity (off-diagonal)
    n = similarity.shape[-1]
    off_diag = similarity - torch.eye(n, device=similarity.device)
    return torch.mean(off_diag ** 2)

Key Insights

1. Continuous vs. Discrete Emotion: Hormone values capture continuous, multi-dimensional emotional states rather than discrete categories (happy/sad/angry)
2. Biological Grounding: Endocrine system provides natural model for slow, modulatory signals that influence cognition — analogous to how hormones affect brain states
3. Temperature Scaling: Allows hormone specificity tuning — high temperature for broad emotional influence, low temperature for specific emotional responses
4. Orthogonal Initialization: Critical for preventing hormone collapse — ensures each hormone dimension learns distinct emotional aspects

Applications

• Emotionally intelligent chatbots: More empathetic, contextually appropriate responses
• Affective NLP: Fine-grained emotion detection and generation
• Therapeutic AI: Mental health applications requiring emotional sensitivity
• Creative writing assistants: Tone-aware content generation
• Bio-inspired AI architectures: Extending to other biological modulation systems

Activation

• hormone-t5, hormone emotion layer, HELT
• emotional language models, affective computing transformer
• endocrine-inspired AI, bio-inspired emotion modeling
• continuous emotion representation, hormone attention
• 情感层, 激素启发, 情感语言模型

Limitations

• Requires curated emotion-labeled training data
• Hormone dimensions are abstract — mapping to real hormones is approximate
• Added computational overhead (per-hormone attention heads)
• Best suited for conversational/affective tasks, not all NLP applications

Related Work

• Discrete emotion classification in NLP
• Sentiment analysis (coarse-grained)
• Biological emotion modeling (affective neuroscience)
• Modulatory mechanisms in neural networks