dataviewer

# Dataviewer Skill Launch and interact with the Dataset Analysis Tool, a full-stack application for analyzing and annotating robotic training data from episode-based datasets. Complete requested operations with persistence read-back; job acceptance is not inference completion or label application. ## Prerequisites | Platform | Requirement | |----------|-----------------------------------------------------------------------------------------------------------------------| | All | Python 3.12+, Node.js 24+, npm, `uv` | | All | `ffmpeg` on `PATH` outside the backend container, for HDF5 camera video (OpenCV also works) and LeRobot episode clips | The backend virtual environment and repository-root npm workspace dependencies are auto-created on first launch by `start.sh`. ## Launch and Connect Workflow Follow these steps when a launch is required and authorized. Inspect existing terminals/task output first and reuse healthy services with matching configuration. Read-only inspection does not authorize restarts, inference or dataset writes. Use VS Code tasks for an already configured development session; use the launcher for requested overrides and readiness checks. When a caller or workflow supplies the URL of a Dataviewer it already runs, such as the instance the Sim Workspace Command Center opens for a workspace, skip Step 1 and open that URL in Step 2. Start `start.sh` only when no running instance is supplied, so one dataset folder never gets a second Dataviewer. ### Step 1 — Start the app Launch `start.sh` as a background terminal process. The script prints `[OK] Both services are running` and the URLs when both services are healthy. Use `./start.sh --check` to check installed launch prerequisites without starting or installing services. The launcher binds both services to loopback, requires the selected frontend port and returns failure when either service fails readiness. ```bash cd data-management/viewer && ./start.sh ``` With a custom dataset path: ```bash cd data-management/viewer && ./start.sh --data-dir /path/to/datasets ``` ### Step 2 — Open the app in the browser After confirming both services are running (look for `[OK] Backend is healthy` in terminal output), or when a running URL was supplied, open the frontend with the integrated browser's `open_browser_page` tool: ```text open_browser_page("http://localhost:5173") ``` SimpleBrowser is the visual interface for the user when that provider is available. Configured headless Playwright automation runs in the background. Native browser tools can operate a shared page directly; separate browser sessions share persisted backend state, not necessarily unsaved drafts. Use the supplied URL, or substitute a non-default `FRONTEND_PORT` for `5173`. ### Step 3 — Load browser automation tools Use the host's tool search to discover available browser capabilities before calling deferred tools. Native tools include `read_page`, `run_playwright_code`, `navigate_page`, `click_element`, `type_in_page` and `screenshot_page`. Use those when available; an equivalent native read does not require MCP reconfiguration. Drive the UI with whichever browser tool family the host provides: | Tool family | Tools | How it works | |--------------------|------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------| | Integrated browser | `open_browser_page`, `navigate_page`, `read_page`, `click_element`, `type_in_page`, `run_playwright_code`, `screenshot_page` | Acts on the page the user sees | | Playwright MCP | `browser_snapshot`, `browser_navigate`, `browser_click`, `browser_type`, `browser_evaluate`, `browser_take_screenshot` | Acts headlessly on a separate copy of the same URL | For the Playwright MCP background workflow, configure **headless mode** so it does not open a separate browser window. Its visual feedback goes through SimpleBrowser (Step 2). A Playwright MCP server configuration in `.vscode/mcp.json` can use: ```json // .vscode/mcp.json { "servers": { "playwright": { "command": "npx", "args": ["@playwright/mcp@latest", "--headless"] } } } ``` > [!IMPORTANT] > Use `--headless` for the MCP background workflow. Reconfiguring or installing a browser provider requires approval; do not change it merely to obtain inspection evidence. Search by capability using the host's available tool-search schema, for example: ```text Find browser tools to read an accessible page snapshot, navigate, click, type and capture screenshots. ``` If no suitable provider is available, report the limit and guide manual inspection. When the user chooses the configured MCP provider, they can run **MCP: Start Server** then select **playwright** in the Command Palette. An empty tool search alone does not prove that a server is stopped. ### Step 4 — Interact with the UI Both tool families see the same backend state, so API-driven changes (labels, annotations) appear in both. | Action | Integrated browser | Playwright MCP | Notes | |--------------------|-----------------------|---------------------------|-----------------------------------------------| | Capture page state | `read_page` | `browser_snapshot` | Inspect current accessible roles/names | | Navigate to URL | `navigate_page` | `browser_navigate` | Preserve unsaved drafts before reload | | Click an element | `click_element` | `browser_click` | Use current snapshot refs and names | | Type into input | `type_in_page` | `browser_type` | For dataset filters or label inputs | | Run a script | `run_playwright_code` | `browser_evaluate` | For sliders, scrolling and multi-step checks | | Take a screenshot | `screenshot_page` | `browser_take_screenshot` | Appearance, not persistence or keyboard proof | Take a fresh accessible snapshot before click/type actions and refresh refs after content changes. Use the [Frontend UI Structure](#frontend-ui-structure) below to orient; forced DOM clicks are not keyboard accessibility evidence. ## Quick Start Start the dataviewer with the default dataset path: ```bash cd data-management/viewer && ./start.sh ``` Start with a custom dataset path: ```bash cd data-management/viewer && ./start.sh --data-dir /path/to/datasets ``` ## Parameters Reference | Parameter | Default | Description | |------------------|----------------------------------------------|---------------------------------------------| | `DATA_DIR` | `../../../datasets` (relative to `backend/`) | Directory containing dataset subdirectories | | `BACKEND_PORT` | `8000` | FastAPI backend port | | `FRONTEND_PORT` | `5173` | Vite frontend dev server port | | `HEALTH_TIMEOUT` | `30` | Seconds to wait for backend health check | ### Dataset Path Configuration The `DATA_DIR` environment variable controls which datasets are visible in the app. Each subdirectory under this path is treated as a separate `dataset_id`. Use one of these methods to set `DATA_DIR`. ### Launch argument Use this method for workflow handoff and ad-hoc sessions: ```bash ./start.sh --data-dir /path/to/datasets ``` ### Backend environment file Use this method only when a persistent local default is requested: ```env DATA_DIR=/path/to/datasets ``` ### Session environment Use this method for a shell-scoped override: ```bash export DATA_DIR=/path/to/datasets cd data-management/viewer && ./start.sh ``` Persist a path in `backend/.env` only when the user explicitly requests a local default across restarts. Do not mutate `.env` for a profile-bound workflow handoff. The workflow owns the viewer child and passes the manifest's exact dataset parent through `--data-dir`. When `accepted-dataset.json` exists, verify the dataset capabilities before browser inspection. Require the returned dataset ID, output adapter/version, viewer adapter, profile ID/hash, capture-provenance hash, export-validation hash, capture features, and sensor roles to match the descriptor. Read the [accepted-dataset contract reference](../../../data-management/viewer/README.md#-accepted-dataset-contract) for the descriptor schema, artifact integrity checks, and invalid-contract behavior. ## Architecture ```text data-management/viewer/ ├── start.sh # Orchestrator: launches backend + frontend ├── backend/ │ ├── .env # DATA_DIR and test config │ ├── pyproject.toml # Python dependencies (uv) │ └── src/api/ │ ├── main.py # FastAPI app, CORS, router registration │ ├── routers/ # REST endpoints: datasets, annotations, labels, export, detection, analysis │ ├── routes/ # AI analysis routes │ ├── services/ # Business logic and dataset service │ ├── models/ # Pydantic models │ └── storage/ # Persistence layer ├── frontend/ │ ├── vite.config.ts # Dev server + API proxy to :8000 │ └── src/ │ ├── App.tsx # Root: dataset selector, episode list, annotation workspace │ ├── api/ # HTTP client and typed API functions │ ├── components/ # UI components (annotation, dashboard, episode viewer, export) │ ├── hooks/ # React Query hooks for datasets, episodes, annotations │ ├── stores/ # Zustand stores for episode and dataset state │ └── types/ # TypeScript type definitions ``` ## API Reference ### Core Endpoints | Endpoint | Method | Description | |----------------------------------------------------|--------|-------------------------------------------------------------------------------------------| | `/health` | GET | Health check | | `/api/datasets` | GET | List all datasets | | `/api/datasets/{id}` | GET | Get dataset metadata | | `/api/datasets/{id}/capabilities` | GET | Get format support, optional feature availability, and verified accepted-dataset contract | | `/api/datasets/{id}/episodes` | GET | List episodes in a dataset | | `/api/datasets/{id}/episodes/{idx}` | GET | Get episode data with trajectory and metadata | | `/api/datasets/{id}/episodes/{idx}/trajectory` | GET | Get trajectory data only | | `/api/datasets/{id}/episodes/{idx}/frames/{frame}` | GET | Get a single frame image | | `/api/datasets/{id}/episodes/{idx}/cameras` | GET | List available camera views | | `/api/datasets/{id}/episodes/{idx}/video/{camera}` | GET | Stream video for a camera | | `http://localhost:8000/docs` | GET | Swagger UI documentation | ### Label Endpoints | Endpoint | Method | Description | |--------------------------------------------|--------|-------------------------------------------------------| | `/api/datasets/{id}/labels` | GET | Get all episode labels and available label options | | `/api/datasets/{id}/labels/options` | GET | List available label options |