sensorless-gaze-following

name	sensorless-gaze-following
description	Neuroscience framework for sensorless gaze-following in Human-Robot Interaction (HRI). Uses brain-inspired prediction mechanisms to estimate gaze targets without eye-tracking hardware.
version	1.0.0
author	Research Synthesis
license	MIT
metadata	{"hermes":{"tags":["neuroscience","gaze-following","hri","brain-inspired","robotics","social-cognition"],"source_paper":"Perception Is All You Need? Neuroscience Framework for Sensorless Gaze-Following in HRI (arXiv:2601.06429)"}}

Sensorless Gaze-Following in Human-Robot Interaction

Overview

This framework leverages neuroscience insights about human gaze-following behavior to enable robots to estimate human gaze targets without requiring eye-tracking hardware. By understanding how the human brain predicts gaze direction from head pose, body orientation, and contextual cues, robots can achieve natural social interaction through sensorless gaze estimation.

Key Insights

1. Brain-Inspired Prediction: Human gaze-following relies on predictive mechanisms in the superior temporal sulcus (STS) and intraparietal sulcus (IPS), which can be computationally modeled
2. Multi-Modal Integration: Gaze estimation integrates head pose, body orientation, scene context, and social priors — not just eye direction
3. Context-Aware Attention: The brain uses scene semantics and task context to disambiguate gaze targets in cluttered environments
4. Social Cognition: Joint attention emerges from shared spatial representations and intention modeling

Core Framework

┌──────────────────────────────────────────────────┐
│           Sensorless Gaze Estimator              │
├──────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌──────────────────────┐      │
│  │  Head Pose  │  │  Body Orientation    │      │
│  │  Encoder    │  │  Encoder             │      │
│  └──────┬──────┘  └──────────┬───────────┘      │
│         │                   │                   │
│  ┌──────▼───────────────────▼──────────────┐    │
│  │       STS-Inspired Fusion Layer         │    │
│  │  (multi-modal attention integration)    │    │
│  └──────────────────┬──────────────────────┘    │
│                     │                           │
│  ┌──────────────────▼──────────────────────┐    │
│  │       IPS-Inspired Spatial Mapping      │    │
│  │  (gaze vector → scene target)           │    │
│  └──────────────────┬──────────────────────┘    │
│                     │                           │
│  ┌──────────────────▼──────────────────────┐    │
│  │     Context & Social Prior Modulator    │    │
│  └──────────────────┬──────────────────────┘    │
│                     │                           │
│  ┌──────────────────▼──────────────────────┐    │
│  │         Gaze Target Prediction          │    │
│  └─────────────────────────────────────────┘    │
└──────────────────────────────────────────────────┘

Implementation Pattern

import numpy as np
from dataclasses import dataclass
from typing import Optional

@dataclass
class GazeEstimate:
    """Gaze target estimation result."""
    target_position: np.ndarray  # 3D coordinates of estimated gaze target
    confidence: float
    target_object: Optional[str]  # Semantic label of target object
    uncertainty_cone: float  # Angular uncertainty in degrees

class SensorlessGazeEstimator:
    """
    Neuroscience-inspired sensorless gaze-following system.
    
    Estimates human gaze targets using:
    - Head pose and body orientation (visual cues)
    - Scene context and object semantics
    - Social interaction priors
    """
    
    def __init__(self, n_objects: int = 10):
        self.n_objects = n_objects
        
        # STS-inspired: multi-modal fusion weights
        self.head_pose_weight = 0.5
        self.body_weight = 0.3
        self.context_weight = 0.2
        
        # IPS-inspired: spatial attention map
        self.attention_map = np.zeros((64, 64))
        
        # Social priors: object salience scores
        self.object_salience = np.ones(n_objects) / n_objects
    
    def () -> GazeEstimate:
        
        
        
        gaze_direction = ._fuse_multimodal_cues(
            head_pose, body_orientation
        )
        
        
        object_scores = ._spatial_attention(
            gaze_direction, scene_objects
        )
        
        
         social_context:
            object_scores = ._apply_social_priors(
                object_scores, scene_objects, social_context
            )
        
        
        target_idx = np.argmax(object_scores)
        confidence = ._compute_confidence(object_scores)
        
         GazeEstimate(
            target_position=scene_objects[target_idx][],
            target_object=scene_objects[target_idx][],
            confidence=confidence,
            uncertainty_cone=._compute_uncertainty(object_scores)
        )
    
     () -> np.ndarray:
        
        
        head_direction = np.array([
            np.cos(head_pose[]) * np.cos(head_pose[]),
            np.sin(head_pose[]) * np.cos(head_pose[]),
            np.sin(head_pose[])
        ])
        
        
        body_direction = np.array([
            np.cos(body_orientation[]),
            np.sin(body_orientation[]),
            
        ])
        
        
        fused = (.head_pose_weight * head_direction +
                 .body_weight * body_direction)
        fused /= np.linalg.norm(fused)
        
         fused
    
     () -> np.ndarray:
        
        scores = np.zeros((scene_objects))
         i, (position, _)  (scene_objects):
            
            obj_dir = position / np.linalg.norm(position)
            alignment = np.dot(gaze_direction, obj_dir)
            scores[i] = np.clip(alignment, , )
         scores
    
     () -> np.ndarray:
        
        task_relevance = context.get(, {})
         i, (_, label)  (scene_objects):
             label  task_relevance:
                scores[i] += task_relevance[label] * .context_weight
         scores / scores.()
    
     () -> :
        
        normalized = scores / scores.()
        entropy = -np.(normalized * np.log(normalized + ))
        max_entropy = np.log((scores))
          - (entropy / max_entropy)  max_entropy >   
    
     () -> :
        
        normalized = scores / scores.()
        spread = np.std(normalized)
         np.clip( * ( - spread * ), , )

Applications

1. Human-Robot Collaboration: Robots that naturally follow human gaze during joint tasks
2. Assistive Robotics: Understanding user intent without wearable sensors
3. Social Robotics: Natural eye contact and joint attention behaviors
4. Smart Environments: Gaze-aware interactive displays and systems
5. Autonomous Vehicles: Predicting pedestrian attention and intent

Key Parameters

Parameter	Description	Typical Range
head_pose_weight	Importance of head pose	0.4 - 0.7
body_weight	Importance of body orientation	0.2 - 0.4
context_weight	Importance of social context	0.1 - 0.3
attention_resolution	Spatial attention map size	32x32 - 128x128

Activation Keywords

• sensorless gaze following
• gaze estimation
• human-robot interaction
• joint attention
• brain-inspired robotics
• social cognition
• head pose gaze
• 无传感器注视跟踪
• 人机交互注视

References

• Original Paper: Perception Is All You Need? Neuroscience Framework for Sensorless Gaze-Following in HRI. arXiv:2601.06429 (2026)
• Related Skills: [[neural-brain-framework]], [[context-selective-multimodal-memory]], [[ember-hybrid-snn-llm-architecture]]

Limitations

• Accuracy degrades with occluded head/face views
• Requires accurate head pose estimation
• Social priors need domain-specific calibration
• Performance depends on scene understanding quality