vs-item-onboarding

# Viking Item Onboarding ## Language Matching (apply throughout) Match the language of the **user's most recent message** in every line of prose you write — confirmation prompts, status notes, hand-off summaries, questions, error explanations, **and any internal thinking / reasoning / planning output that the host may surface (e.g. `<thinking>` blocks, "thinking" panels, scratchpad notes, todo descriptions)**. If the user is writing in Chinese, every prose line and every reasoning line must also be in Chinese; if English, English; same for Japanese, etc. The fact that this skill file is written in English is for documentation only — at runtime translate all **prose and reasoning** into the user's language. Do not switch back to English mid-flow just because the surrounding skill text is English. **Chinese-user priority (the most common case)** — when `current_query` or the most recent user message is in Chinese: - All prose you write for the user (confirmation prompts, status notes, error explanations, final hand-off summaries) must be in Chinese. - All internal thinking / reasoning / planning output (thinking blocks, scratchpad, todo descriptions) must also be in Chinese. - For workspace artifacts you create, the description / comment portions (excluding CLI-contract English identifiers) should also prefer Chinese. Do **not** translate the following — keep them verbatim so the contract stays machine-checkable: - The verbatim CLI block between `<!-- vs-schema-confirm: BEGIN -->` and `<!-- vs-schema-confirm: END -->` (English section labels `**Metadata**` / `**Fields (N)**` / `**Field Roles**` / `**Warnings (N)**` and English warning text come straight from the CLI). - CLI command names, flag names, JSON keys, enum values, field names, primary-key BizAttr identifiers (`multi_modal_id`), dataset IDs / app IDs / TaskIDs, and console URLs. - The single literal token the user must reply to confirm — write it as `` `yes` `` in any language so the contract for advancing to step 8 is unambiguous (you may add a parenthetical native-language hint, e.g. `回复 \`yes\`（即"确认"）继续`). If you are unsure which language the user used (e.g. only emoji or only an attachment), default to the language of the very first user turn in the conversation. When the user switches languages mid-flow, switch with them on the next message. ## When to Use Use this skill when the user is operating against the V2 control-plane (`/open/*V2`) and wants to onboard a dataset (optionally followed by an application) from either: - a local `JSONL` file, either as a one-time import or with ongoing incremental sync for append-only files (e.g. crawler output where new lines are continuously appended); - a local `JSON` (array) or `CSV` file, as a one-time import only (ongoing sync is not supported for these formats); - a MySQL database, either as a one-time snapshot import or with ongoing incremental sync. Supported dataset types: - **`user_event`** — user behavior / event logs (click, view, exposure, collect, etc.) used for recommendation and personalization scenarios. - **`multi_modal`** — everything else: records that contain image URLs and/or video URLs alongside text fields (e.g. e-commerce goods with images, short-video posts, content with thumbnails, multimodal search corpora), as well as plain-text corpora without media. If the data is not behavior logs, it goes here. The hallmark of V2 is that schema inference is fully backend-driven: the CLI uploads the file, the backend infers the `Schema` (with `BizAttr` already set on the primary-key / title / URL / event-type fields) plus a per-field `FieldDescMap`, and the agent's only jobs are to (a) persist that inference artifact locally, (b) render it for one round of human confirmation, and (c) drive the remaining persistence + ingest steps without re-inventing field decisions. For parent/variant item datasets, schema inference may assign paired ItemType / ParentId BizAttrs such as `ImageItemType` + `ImageParentId` or `multi_modal_item_type` + `multi_modal_parent_id`. These fields must be treated as a pair: ItemType identifies parent vs variant/child items, and ParentId points a variant/child item to its parent. If the Schema Confirmation warnings report that only one side was inferred, do not patch BizAttrs by hand; fix the source data or field meanings and re-run inference. Do not use this skill when: - The customer only wants to ingest more rows into an existing dataset (use `vs data write --dataset-id <id> --fields @items.jsonl`; `--fields` accepts a JSON array or a JSONL file directly). ## Do NOT be misled by `vs --help` top-level QUICK START `vs --help` still lists `vs item profile / plan / apply` at the top of QUICK START for backwards compatibility (annotated `[Deprecated]`). That is the V1 path; this skill does **not** use it. The only legal path here is V2 — `vs dataset import-url → infer-schema → infer-result → dataset create → data write → app create → app attach-dataset` — and any check for a V2 command must be confirmed via `vs dataset --help`, `vs dataset infer-schema --help`, `vs app --help`, or `vs app attach-dataset --help`, never by falling back to `vs item ...`. The workspace path `./.viking/item-plans/<dataset-name>/` is reused for V2 artifacts only because the directory name happens to match; it does not imply V1 or item type. The moment the user's ask is "create a multi-modal dataset / application from a raw JSONL / JSON / CSV / MySQL source", jump straight to the V2 workflow (steps 3–14 below) without detouring through `item plan/apply`. **Forbidden in this skill:** passing any `--type` other than `multi_modal` or `user_event` to dataset onboarding commands. ## Version Check Before starting this skill workflow, run `vs version check --json`. Continue only when `status` is `up-to-date`. If `status` is `update-available`, stop and tell the user to update the cloned `vs` repository, then run `git pull --ff-only`, `bash ./scripts/install.sh`, and `bash ./scripts/install-skills.sh all --target auto --force` (PowerShell: `scripts/install.ps1` and `scripts/install-skills.ps1`). If the status is `unknown`, stop and report that the CLI version could not be verified. ## Preconditions - `vs` CLI ≥ 0.2.0 installed, authentication is complete (`vs auth status` and `vs doctor` succeed). - Input file is `JSON array`, `JSONL`, or `CSV` and is readable from a local path. - The user has stated a business goal (e.g. "Build catalog search", "Build content search"). - The customer's account is provisioned for the V2 control-plane. ## Commands | Stage | CLI command | Purpose | |---|---|---| | Upload URL | `vs dataset import-url --file-name <basename>` | Request a presigned PUT URL plus `FileKey` | | PUT upload | `curl -X PUT --data-binary @<path> "<FileUrl>"` | Upload the local file to TOS (no auth header needed) | | Submit inference | `vs dataset infer-schema --tos-key <FileKey> --type <multi_modal\|user_event> [--theme <general\|e_commerce\|content\|long_video>] --language <zh\|en\|ko\|ja\|hi> [--name ...]` | Kick off backend schema inference; returns `TaskID`. `--theme` is **required for `multi_modal` only** (default `general`); omit it for `user_event`. The CLI accepts theme aliases such as `ecommerce` / `e-commerce` → `e_commerce`, `long-video` / `longvideo` → `long_video`, `common` / `default` → `general`. | | Poll inference | `vs dataset infer-result --task-id <TaskID>` | Poll until `Status=succeeded`; returns `Schema` + `DataFieldConfig` (the entire inference artifact). For `multi_modal`, includes `ImageIndexFields` / `VideoIndexFields` / `ChatFields`. | | Validate schema | `vs dataset validate-schema --input <path/to/infer-result.json> --dataset-type <multi_modal\|user_event>` | Render the deterministic schema-confirm block (metadata / fields / roles / warnings). Use `--dataset-type` to toggle validation rules. Save the output as the source-of-truth for schema confirmation. | | Create dataset | `vs dataset create --data @dataset-create.json [--post-paid-type <standard\|premium>] [--dry-run]` | Persist (or dry-run) the inferred schema. **Do not** flip `IsPK` — backend derives PK from `BizAttr`. For `multi_modal`, the payload must include `Theme` and optionally `ProcessConfig`; for `user_event`, omit both. Pass `--post-paid-type` only for post-paid billing instances (`standard`/`premium`); omit it for non-post-paid (`none`) instances. | | Write data | `vs data write --dataset-id <DatasetId> --fields @items.jsonl` | Push the actual records into the dataset. `--fields` accepts a JSON array **or** a JSONL file (one record per line); the bootstrap `items.jsonl` can be passed directly — no need to convert with `jq -s`. | | Export (MySQL) | `vs connector export --source mysql ...` | Export a MySQL table snapshot into `/tmp/viking/connector/<job>/bootstrap/items.jsonl` | | Export (local file) | `vs connector export --source jsonl --file <path>` | Export a local JSONL file snapshot into the bootstrap directory. For `JSON` (array) or `CSV` inputs, convert to JSONL (one object per line) before running this command. | | Sync config | `vs connector init --name <job> --source mysql\|jsonl --dataset-id <id> ...` | Persist the local sync job config for later incremental runs | | Sync run | `vs connector run --job <job> --daemon` | Start background incremental sync into the dataset | | Create application | `vs app create --name <name> --industry <industry> --language <lang> [--description ...] [--color cyan\|blue\|purple\|pink] [--risk-check] [--post-paid-type <standard\|premium>] [--dry-run]` | Optional, only when the user asks for app-level setup. `--industry` here is an **application-level** attribute independent of the dataset; it is NOT passed to dataset create / infer-schema. Pass `--post-paid-type` only for post-paid billing instances (`standard`/`premium`); omit for non-post-paid (`none`). | | Attach dataset | `vs app attach-dataset --data @attach.json [--dry-run]` | Optional, links the created dataset to an application. The `DataConfig` block is the `DataFieldConfig` straight out of the persisted infer artifact (must include `ImageIndexFields` / `VideoIndexFields` / `ChatFields` verbatim) | The "All-in-one" shortcut `vs dataset ingest --file <path> --type multi_modal --theme <theme> [--abnormal-image-policy skip|block] [--abnormal-video-policy skip|block] [--video-auto-delete] [--post-paid-type <standard|premium>] [--dry-run]` orchestrates upload + infer-schema + poll + create + write, **without** the Schema Confirmation pause. In agent mode you should still drive each step individually so you can pause at step 7 (Schema Confirmation). ## Workflow Run strictly in order. Each step depends on output from the previous one; an inference artifact persisted in step 6 is reused all the way through step 13. 1. **Confirm dataset type, input source, and mode** — first determine the dataset type, then identify the source type, then **explicitly ask the user to choose the import mode** when multiple options exist. Do not silently pick any of these. **Dataset type resolution** — ask the user which dataset type they want to create: - **`multi_modal`** — records with image URLs and/or video URLs plus text fields (e-commerce goods, short-video posts, content with thumbnails, etc.). - **`user_event`** — user behavior / event logs (click, view, exposure, collect, etc.) for recommendation and personalization. If the user's request clearly describes behavior logs / event data / recommendation data → `user_event`; if it clearly describes goods / content with images or video → `multi_modal`; if ambiguous, **ask**. **For `multi_modal` only — Theme resolution (mandatory)** — the backend requires a valid `Theme`. Ask the user to pick one: - `e_commerce` — e-commerce products (with images, price, brand, tags) - `long_video` — long-form video (movies, series; with cover image, language, category) - `content` — general short-form content (posts, news articles with thumbnails, tags, categories) - `general` — other / generic multi-modal (default) If the user cannot decide, default to `general`. Record the chosen theme in a local variable and pass it to every subsequent command that accepts `--theme`. **Source identification**: - If the user provided a database connection or table name → MySQL. - If the user provided a file path ending in `.jsonl` or described a line-delimited/append-only file → JSONL. - If the user provided a file path ending in `.json` (JSON array) or `.csv` → JSON/CSV (one-time only). - If the source is unclear, ask the user which source type they want to onboard from before proceeding. **Language**: ask for language if the user has not already stated it (`zh` / `en` / `ko` / `ja` / `hi`); default to `zh` for Chinese-speaking users, `en` otherwise. **Import mode selection**: - For **MySQL** and **JSONL**, resolve whether the user wants one-time import or one-time + ongoing sync. **Only skip the question when the request contains an explicit, unambiguous signal** for one side (apply this detection to whatever language the user is writing in — English, Chinese, etc.): - **Explicit one-time**: phrases carrying "once", "one-time", "snapshot only", "just this time", or equivalent single-import semantics. - **Explicit ongoing**: phrases carrying "sync", "keep in sync", "auto-import", "scheduled", "incremental", "keep updated", or equivalent recurring-sync semantics. - If the request is **neutral** — e.g. "import this file", "import this data", bare "import", mentions only a file path with an import verb but says nothing about scheduling/increment/once — **you MUST ask the user to choose**. The bare import verb is NOT a one-time signal; it is ambiguous. **Never silently default to one-time.** - For **JSON (array)** and **CSV**, only one-time import is supported. No question needed. After the dataset type, source type, theme (if multi_modal), language, and mode are confirmed, follow the matching branch: - **MySQL — one-time import**: identify the table name, infer dataset name and primary key, and require explicit confirmation before any real write. Continue at step 2. - **MySQL — ongoing sync**: same as above, plus the user must explicitly confirm the incremental cursor field itself. After step 10 continue at step 11. - **MySQL — existing dataset — ongoing sync**: validate the dataset with `vs dataset get --id <DatasetId> --full`, confirm the source config (especially the incremental cursor field), then jump directly to step 11. - **JSONL file — one-time import**: confirm the file path. Continue at step 2. - **JSONL file — ongoing sync**: confirm the file path. You MUST also **interactively ask the user to confirm** that new records will only be appended to the end of the file (append-only). Present the constraint clearly — sync only supports files that grow by adding new lines; edits or deletions of existing lines are not tracked and may cause duplicate or missing records. Wait for explicit user confirmation before proceeding. After step 10 continue at step 11. - **JSON (array) or CSV file — one-time import**: confirm the file path. These formats are one-time import only; ongoing sync is not supported because they do not provide a stable append-only cursor. Convert the input to JSONL (one JSON object per line) before continuing. Continue at step 2. - **Existing dataset + one-time source import**: not supported as a single workflow. Explain that the current CLI split supports either source export → new dataset onboarding for a one-time import, or background sync for ongoing updates, then let the user choose which branch to switch to. Source environment configuration (applies to MySQL branches only; local files require no credentials): - MySQL uses these environment variables by default: `MYSQL_HOST`, `MYSQL_PORT`, `MYSQL_USER`, `MYSQL_PASSWORD`, `MYSQL_DATABASE` (optional: `MYSQL_CHARSET`). - Render a **bash export template snippet** with placeholder values (for example `MYSQL_PASSWORD=your_password`) and ask the user to fill in real values in their own terminal or shell session, then run `export` on each variable. - **Never** display actual database credential values in chat. **Never** ask the user to paste or submit database credentials into the chat dialog. - **Never** list "connection config" / "连接配置" blocks with concrete host/user/password values inside the chat. The only allowed format is a bash template with placeholder values. - The export, init, and run commands read MySQL credentials **only from environment variables**. They do not accept credentials via flags or chat input. - This is a human checkpoint. Wait for explicit confirmation that the local source environment is configured before proceeding. 2. **Export source snapshot to JSONL** — run `vs connector export` for the selected source to produce a bootstrap JSONL file: - MySQL: `vs connector export --source mysql --source-table <table> --id-field <field> --cursor-field <field> [other flags]` - Local file: `vs connector export --source jsonl --file <path/to/items.jsonl> [other flags]` (convert JSON arrays or CSV to JSONL first if needed) The bootstrap file is always written to `/tmp/viking/connector/<job>/bootstrap/items.jsonl`. Do not use `--output` to try to override that path; `--output` only redirects the rendered command result. After export, use the emitted `items.jsonl` as the input file and continue at step 3. 3. **Get upload URL** — `vs dataset import-url --file-name <basename>`. Capture `Result.FileUrl` and `Result.FileKey`. Keep `FileKey` for step 5. 4. **PUT upload** — upload the raw item file to `FileUrl` (e.g. `curl -X PUT --data-binary "@<local-path>" "<FileUrl>"`). Expect HTTP 200 with empty body. Do not add an `Authorization` header — `FileUrl` is already presigned.