| name | tao-generate-referring-expressions |
| description | Four-step image referring-expression pipeline: turns images plus KITTI bounding-box labels into region descriptions, scene captions, grounded referring expressions, and (optionally) verified expressions via VLM distillation. Use when the user wants to generate referring-expression annotations from images with KITTI labels, build region descriptions, produce grouped grounding phrases tied to bboxes, run a double-check verification pass on grounding expressions, auto-label traffic / scene images for referring datasets, or run the image_referring_expression pipeline. Triggers include 'referring expression', 'region description', 'KITTI labels', 'spatial relationship annotation', 'auto-label image referring expression', 'image_referring_expression'. |
| license | Apache-2.0 |
| compatibility | Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible). |
| metadata | {"author":"NVIDIA Corporation","version":"0.1.0"} |
| tags | ["image","referring-expression","kitti","bounding-boxes","auto-label","vlm"] |
| allowed-tools | Read Bash Write |
Image Referring Expression Pipeline
Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).
Generate referring-expression and grounding annotations from images with KITTI-format bounding box labels. A single VLM (Gemini or any OpenAI-compatible endpoint) runs four steps: per-object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double-check verification pass.
Purpose
Transform (image, KITTI labels) pairs into a unified annotations.jsonl containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0-1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re-examines those bboxes against the image and corrects mismatches.
Pipeline Architecture
Step 0: Region expression ──┐
├──▶ Step 2: Grounding expression ──▶ [Step 3: Double check]
Step 1: Image caption ──────┘ (optional)
- Step 0 (region_expr) — VLM emits one short discriminative phrase per KITTI bbox (
bbox_2d, type, color, description).
- Step 1 (image_caption) — VLM emits a holistic, location-agnostic scene caption.
- Step 2 (grounding_expr) — VLM groups Step 0 objects into grouping phrases and returns one bbox list per group, optionally using Step 1's caption as extra context.
- Step 3 (double_check) — VLM re-checks each Step 2 bbox against the image; bad matches are removed, slightly-off boxes get tightened.
Steps 0 and 1 run in parallel within a single thread pool (they only depend on the seed records). Each step writes its own step_<N>_*/annotations.jsonl and skips already-processed images on re-run unless workflow.force_reprocess: true.
Instructions
Initial setup
When a user wants to run this pipeline, walk through these steps:
-
Images: Ask for data.image_dir, the directory containing .jpg, .jpeg, or .png images.
-
KITTI labels: Ask for data.kitti_label_dir, the directory containing one label file per image. Each label line must use KITTI format: . Lines with fewer than 8 fields are silently skipped. Set this even for Step 1-only runs because Steps 0 and 2 require it.
