| name | pearl-personalized-streaming-video |
| title | PEARL: Personalized Streaming Video Understanding — Problem Definition and Framework |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.20422 |
| keywords | ["Personalized Video Understanding","Streaming Video","Real-Time Interaction","Video-Level Concepts","Frame-Level Concepts"] |
| description | Defines Personalized Streaming Video Understanding (PSVU) as a new task bridging static image personalization and video intelligence. PEARL-Bench (132 videos, 2173 annotations) establishes founding experiments. PEARL framework provides training-free plug-and-play strategy using dual-grained memory (concept + streaming) and concept-aware retrieval. Trigger: When building AI assistants that interact with streaming video and personalize on user-defined concepts, apply this problem definition and framework to support real-time multi-turn reasoning. |
| category | Field Foundation |
The Problem Statement
What is this problem?
Current AI personalization operates on static images or offline video, disconnecting it from real-world scenarios requiring continuous visual input and instant feedback. Users watch livestreams, security cameras, or personal recordings and expect AI assistants to understand:
- Concepts specific to their use case (custom objects, people, actions)
- Real-time events unfolding in the stream
- Historical context from earlier stream segments
No prior problem definition or benchmark existed for this scenario.
Why is this important?
Streaming video personalization is ubiquitous in practice: live event interpretation, security monitoring, personal video archives, accessibility assistance. Yet research focused on static image personalization, missing the unique challenges of temporal continuity and real-time interaction.
What existing approaches are inadequate?
- Static image personalization: Works on single frames, can't reason about temporal dynamics or streaming context
- Offline video understanding: Analyzes complete, pre-recorded video; doesn't handle streaming constraints or personalization
- Generic video captioning: No mechanism for user-defined concepts or real-time interaction
These approaches lack the infrastructure for continuous concept registration, temporal reasoning, and interactive multi-turn dialogue.
The New Paradigm: Personalized Streaming Video Understanding (PSVU)
What new problem class does PEARL introduce?
PSVU is a distinct research area at the intersection of personalization, streaming inference, and video understanding. It defines three core task components:
- Concept-Definition QA: Register new user-defined concepts (people, objects, actions) from video frames
- Real-Time QA: Answer queries about immediate moments in the stream
- Past-Time QA: Retrieve and reason about historical stream segments
Key terminology:
- Frame-level concepts: Static entities registered from single frames (specific persons, objects visible in a moment)
- Video-level concepts: Dynamic actions unfolding across time (a person walking, object being manipulated)
- Streaming memory: Running buffer of observed frames and their embeddings
- Concept memory: Repository of user-defined concepts and their representations
