multimodal-dataprep-user

# Multimodal DataPrep — User Run deployment and API commands when authorized, then report their actual output. API base: `http://localhost:6007/v1/dataprep`. Multimodal DataPrep creates embeddings and metadata for retrieval. It does not offer a vector-query/search endpoint. ## Load project resources as needed Paths are relative to `microservices/visual-data-preparation-for-retrieval/multimodal-dataprep/`. | Resource | Read when | |---|---| | `docs/user-guide/api-reference.md` and `docs/user-guide/_assets/openapi.yaml` | Constructing media, image, batch, live-stream, summary, download, delete, or telemetry requests | | `docs/user-guide/get-started.md` | Configuring devices, detection, batching, duplicate policy, or environment variables | | `docs/user-guide/pluggable-backends.md` | Selecting VDMS/Milvus or MinIO/local and diagnosing backend behavior | | `docs/user-guide/telemetry-metrics.md` | Reading ingestion telemetry or configuring Metrics Manager | | `setup.sh` and `docker/compose*.yaml` | Deploying a stack | Example scenarios: | Example | Purpose | |---|---| | [manufacturing-inspection-archive.md](./example-prompts/manufacturing-inspection-archive.md) | Ingest object-aware production media | | [object-aware-video-catalog.md](./example-prompts/object-aware-video-catalog.md) | Build retrieval-ready frame and crop records | | [edge-video-preprocessing-box.md](./example-prompts/edge-video-preprocessing-box.md) | Batch-ingest a mounted edge directory | ## 1. Select deployment backends | Vector backend | Media storage | Compose files | |---|---|---| | VDMS (default) | MinIO (default) | `docker/compose.yaml` | | Milvus | MinIO | `docker/compose-milvus.yaml` | | VDMS | Local filesystem | `docker/compose.yaml` then `docker/compose.storage-local.yaml` | Use `MM_DATAPREP_VECTORDB_BACKEND` (`vdms` or `milvus`) and `MM_DATAPREP_STORAGE_BACKEND` (`minio` or `local`) for custom deployments. Keep the service, retriever, and collection naming consistent. ## 2. Obtain deployment files In a repository checkout, run from the microservice root. Without a checkout, fetch the setup script and the compose file(s) for the selected backend: ```bash RAW='https://raw.githubusercontent.com/open-edge-platform/edge-ai-libraries/main/microservices/visual-data-preparation-for-retrieval/multimodal-dataprep' mkdir -p multimodal-dataprep/docker cd multimodal-dataprep curl -fsSLo setup.sh "$RAW/setup.sh" curl -fsSLo docker/compose.yaml "$RAW/docker/compose.yaml" ``` Also fetch `docker/compose-milvus.yaml` or `docker/compose.storage-local.yaml` when selected. ## 3. Start the prebuilt stack Never commit credentials. For the default VDMS + MinIO deployment: ```bash export MINIO_ROOT_USER='<user>' export MINIO_ROOT_PASSWORD='<strong-password>' export EMBEDDING_MODEL_NAME='CLIP/clip-vit-b-32' export REGISTRY_URL='docker.io/intel' export TAG='latest' source ./setup.sh --nosetup docker compose -f docker/compose.yaml up -d --no-build ``` For Milvus, replace the compose file with `docker/compose-milvus.yaml`. For local media storage, layer the storage override after `docker/compose.yaml`. `setup.sh` must be sourced because it exports Compose variables. With no argument it only exports defaults; it does not start containers. ## 4. Verify readiness and dependencies Wait for the in-process embedding client: ```bash until curl -fsS http://localhost:6007/v1/dataprep/health \ | python3 -c 'import json,sys; d=json.load(sys.stdin); raise SystemExit(0 if d.get("status") == "ok" and d.get("embedding_client_status") == "preloaded" else 1)' do sleep 10 done ``` The current fields are `embedding_client_status`, `model_name`, `embedding_device`, `use_openvino`, `detection_model`, and `detection_device`. Check dependencies separately: ```bash docker compose -f docker/compose.yaml ps curl -fsS http://localhost:6010/minio/health/live timeout 2 bash -c '</dev/tcp/localhost/6020' # default VDMS ``` For Milvus, use the Milvus compose file and probe `localhost:19530`. DataPrep's health response is not a substitute for checking every dependency. ## 5. Ingest supported media Upload a video: ```bash curl -fsS -X POST \ 'http://localhost:6007/v1/dataprep/media/upload?frame_interval=15&enable_object_detection=true&tags=camera-1' \ -F 'file=@/path/to/video.mp4;type=video/mp4' ``` Upload an image through the same multipart endpoint: ```bash curl -fsS -X POST \ 'http://localhost:6007/v1/dataprep/media/upload?enable_object_detection=true&tags=inspection' \ -F 'file=@/path/to/image.jpg;type=image/jpeg' ``` For inline base64 or remote HTTP(S) images, use `POST /media/ingest`. For media already in the selected storage backend, use `POST /media/process`. Asynchronous workflows: - `POST /media/upload/batch` - `POST /media/process/batch` - `POST /media/ingest/batch` - `POST /media/ingest-dir` (also `store_copy: false` to embed files in place without copying them into storage — such media is still listed by `GET /media` with `"stored": false` and streamable via `GET /media/download` — and `metadata` / `meta/<basename>.json` sidecars for user-defined filterable fields) - `GET /media/jobs/{job_id}` - `DELETE /media/jobs/{job_id}` to request cancellation Clean-up: `DELETE /media/{bucket_name}/{video_id}` removes one item, `DELETE /media/{bucket_name}` clears a whole bucket (storage + embeddings). Read the API reference for exact request schemas and configured batch limits. ## 6. Ingest a live RTSP stream A live camera is registered as a resource, not a request: the call returns at once and a background worker keeps ingesting until the stream is paused or deleted. ```bash # Register and start ingesting curl -fsS -X POST 'http://localhost:6007/v1/dataprep/media/streams' \ -H 'Content-Type: application/json' \ -d '{"stream_url": "rtsp://camera-1.local:554/stream1", "stream_name": "lobby-cam", "tags": ["lobby"]}' # Watch progress (state, frames processed, reconnects, last error) curl -fsS 'http://localhost:6007/v1/dataprep/media/streams' # Pause / resume without losing the registration curl -fsS -X PATCH 'http://localhost:6007/v1/dataprep/media/streams/<stream_id>' \ -H 'Content-Type: application/json' -d '{"state": "paused"}' # Stop and deregister (add ?purge_embeddings=true&purge_media=true to erase data) curl -fsS -X DELETE 'http://localhost:6007/v1/dataprep/media/streams/<stream_id>' ``` Notes: - Live embeddings and recorded media use `bucket_name` = `MM_DATAPREP_LIVE_STREAM_BUCKET` (default `live-streams`) and `video_id` = `stream_id`, so `GET /media`, `GET /media/download`, and `DELETE /media/{bucket_name}/{video_id}` work on live data too. - Credentials in an RTSP URL are used to connect but never returned or logged. - Live ingestion grows the index forever unless `MM_DATAPREP_LIVE_RETENTION_HOURS` is set. - `MM_DATAPREP_LIVE_STREAM_MAX_CONCURRENT` (default `8`) caps simultaneous streams; further registrations return `503`. - Purging a stream's embeddings or media is destructive — confirm first. ## 7. Add a text summary Use the `video_id` returned by media listing/job results and its bucket: ```bash curl -fsS -X POST 'http://localhost:6007/v1/dataprep/summary' \ -H 'Content-Type: application/json' \ -d '{ "bucket_name": "video-summary", "video_id": "dp_video_1730000000", "video_summary": "forklift narrowly misses pedestrian", "video_start_time": 33, "video_end_time": 41, "tags": ["safety"] }' ``` The selected model must support text embeddings. ## 8. Manage and observe ```bash curl -fsS 'http://localhost:6007/v1/dataprep/media' curl -fsS 'http://localhost:6007/v1/dataprep/telemetry?limit=5' curl -L 'http://localhost:6007/v1/dataprep/media/download?video_id=dp_video_1730000000' \ -o media.bin ``` `GET /media/download` supports HTTP Range requests for seeking. Deletion removes the entire media directory and its matching vectors: ```bash curl -fsS -X DELETE \ 'http://localhost:6007/v1/dataprep/media/video-summary/dp_video_1730000000' ``` There is no current single-file `video_name` deletion option. Deletion is destructive, so obtain explicit confirmation before running it. Set `MM_DATAPREP_METRICS_MANAGER_URL` to publish completed-pipeline `dataprep_embeddings_per_second` values asynchronously. `/telemetry` remains the direct source for detailed per-ingestion stage timings. ## Troubleshooting | Symptom | Action | |---|---| | API is unavailable | Inspect `docker compose ... ps` and DataPrep logs; initial model/YOLOX downloads can take time | | `embedding_client_status` is `not_loaded` or `error` | Verify `EMBEDDING_MODEL_NAME`, model compatibility, device access, and startup logs | | Dimension mismatch | Use a fresh collection compatible with the selected model; never wipe an existing collection without confirmation | | Milvus connection failure behind a proxy | Use `docker/compose-milvus.yaml` and ensure Milvus/etcd/in-cluster addresses bypass proxies | | MinIO authentication failure | Verify the effective `MM_DATAPREP_MINIO_*` values and reuse the same credentials across restarts | | Large upload returns 413 | Identify the rejecting proxy/server from headers and logs; stage media in storage and call `/media/process` | | Duplicate upload returns 409 | `MM_DATAPREP_ALLOW_DUPLICATE_UPLOADS=false` is enforcing content-hash deduplication | | Object crops are absent | Check whether YOLOX downloaded and whether detection is enabled on a supported device | | Local directory ingest is rejected | Keep `dir_path` beneath `MM_DATAPREP_INGEST_DATA_ROOT`; traversal outside that root is intentionally blocked | | Live stream registration returns 503 | The concurrency limit is reached (`MM_DATAPREP_LIVE_STREAM_MAX_CONCURRENT`) or live ingestion is disabled | | Live stream sits in `reconnecting` or `error` | Read `last_error` from `GET /media/streams/{stream_id}`; check camera reachability from inside the container and the reconnect budget | | Live streams vanish after a restart | Ensure the PostgreSQL registry (`MM_DATAPREP_LIVE_STREAM_DB_*`) is reachable and its data volume is persisted |