behavior-vlm-neuroscience

name	behavior-vlm-neuroscience
version	v1.0.0
last_updated	2026-05-11T00:00:00.000Z
description	Finetuning-free behavioral understanding framework for neuroscience using vision-language models. Enables pose estimation and behavioral analysis linking neural activity to natural actions without human annotation. Use when: analyzing animal behavior from video, building neuroscience behavioral pipelines, or doing finetuning-free VLM behavioral understanding.

BehaviorVLM: Behavioral Understanding for Neuroscience

Unified finetuning-free behavioral understanding with vision-language reasoning for neuroscience research.

Problem

Neuroscience requires linking neural activity to natural behavior, but:

• Human annotation doesn't scale
• Unsupervised pipelines are unstable
• Pose estimation and behavioral understanding are separate tasks

Solution Architecture

1. Vision Encoder: Extract visual features from behavioral videos
2. Language Reasoning: Use VLM for semantic behavioral understanding
3. Neural Linking: Connect behavioral outputs to neural recordings
4. No Finetuning: Leverage pretrained VLM zero-shot capabilities

Workflow

1. Input: Freely moving animal video + neural recording data
2. VLM processes video frames for behavioral description
3. Zero-shot behavioral classification and pose estimation
4. Link behavioral features to neural activity patterns
5. Output: Scalable behavioral-neural correlation analysis

Key Advantages

• Scalable: No human annotation required
• Unified: Single model for pose + behavior
• Finetuning-free: Uses pretrained VLM reasoning
• Neuroscience-ready: Direct neural-behavioral linking

Resources

• Paper: https://arxiv.org/abs/2603.12176
• Authors: Jingyang Ke, Weihan Li, Amartya Pradhan

Activation Keywords

• behavioral understanding
• neuroscience VLM
• pose estimation behavior
• neural activity behavior
• 行为分析
• animal behavior analysis
• brain behavior mapping