metis-catalog-ingest

name

metis-catalog-ingest

description

Stage a HuggingFace (or S3) MODEL or training DATASET into the demo Metis catalog so its weights/files land in SeaweedFS in the platform's standard, reusable format — the same store the built-in models (57) and datasets (56) use, and the fast path (stages even 122B models). Use this instead of hand-downloading (cluster→HF egress is ~0.7–3MB/s and ephemeral downloads are not reusable). Models: any type (LLM/VL/ diffusion). Datasets: HF source OR custom file upload. Auth = scripts/demo/demo_api.py cookie login. Use when "모델/데이터셋 스테이징", "HF 모델·데이터 등록", "카탈로그에 추가", "모델/데이터 인제스트", "SeaweedFS에 올려줘", "학습 데이터 추가", "register or stage a HF model/dataset on Metis", "add model or dataset to catalog", "ingest weights or training data". Do NOT use to serve/deploy a model (image → metis-image-gen-serving; LLM fine-tune/serve → metis-finetune-eval) or for IAM/access (metis-iam-access-grant).

Metis Catalog Ingest (models + datasets)

Register a model or dataset with source_type set → the platform ingests the HF/S3 source into SeaweedFS (global/models/ for models; the dataset store for datasets) in the reusable format the built-in catalog uses. Models & datasets share the SAME mechanism. Verified live 2026-07-22.

Step 1: Detect auth + project

cd "$(git rev-parse --show-toplevel)"
[ -f scripts/demo/demo_api.py ] && echo "demo_api OK" || echo "MISSING"

from demo_api import Client; c=Client("demo"); c.login() (cookie; creds in .secrets/). Project id: c.get("/v1/metis/projects")[1]["data"] (ai-platform = e3885a23-…).

Step 2: See what's already staged (reuse first — don't re-download)

models   = c.get(f"/v1/metis-hub/projects/{PID}/model-catalog/?page_size=100")[1]["data"]    # 57
datasets = c.get(f"/v1/metis-hub/projects/{PID}/dataset-catalog?page_size=100")[1]["data"]    # 56
# datasets incl. Wikipedia-Korean-QnA, Wikimedia-Wikipedia-Korean, Telecom/Security/Realestate/
# Manufacturing/Marketing/E-Commerce… — check status "enable"/"available" before staging a new one.

Step 3: Ingest from HuggingFace (⚠️ metis-hub prefix + TRAILING SLASH)

# MODEL
c.post(f"/v1/metis-hub/projects/{PID}/model-catalog/", body={
  "model_id":"<org>/<repo>","model_name":"<name>","source_type":"huggingface",
  "source_uri":"<org>/<repo>","revision":"main"})                       # → 202 {job_id, PENDING}
# DATASET (same shape, dataset-catalog)
c.post(f"/v1/metis-hub/projects/{PID}/dataset-catalog/", body={
  "dataset_id":"<name>","name":"<name>","source_type":"huggingface",
  "source_uri":"<hf-dataset-repo>","revision":"main"})

• source_type ∈ {huggingface, s3} only (git rejected). Presence of source_type triggers ingest.
• Both routes: metis-hub prefix (not metis-fabric/ai-platform) AND a trailing slash on POST (Fiber strict routing → 404 otherwise). GET (no slash) lists.

Step 4: Add a CUSTOM dataset by file upload (no HF source)

Resumable multipart under .../projects/{PID}/datasets/upload: POST /upload/initiate {filename,...} → GET /upload/{upload_id}/status → POST /upload/{upload_id}/complete (→ 202 async). Cancel: DELETE /upload/{upload_id}. Formats via GET /datasets/sample-templates (SFT/DPO/GRPO/CPT JSONL + HF Arrow). Custom files land in the same dataset store, reusable next time.

Step 5: Monitor to available

m = c.get(f"/v1/metis-hub/projects/{PID}/model-catalog/{ID}")[1]     # or dataset via GET .../datasets/{id}
m["status"], m.get("ingest_status"), m.get("s3_path")                # downloading → available; s3_path set

Trust the enriched ingest_status — the /{id}/jobs list can return 0 for a non-admin role. No byte-level progress is exposed. Retry/cancel: POST .../model-catalog/{id}/jobs/{job_id}/{retry|cancel}.

Step 6: Report

State id, ingest_status, and (when done) the SeaweedFS path. Downstream: models serve via metis-image-gen-serving (image) / metis-finetune-eval (LLM); datasets feed TrainJobs / ML Studio.

Gotchas

• Route = metis-hub prefix + trailing slash on POST (both model- and dataset-catalog). Either wrong → 404.
• s3_path null until staged; poll ingest_status. Staging runs on a platform service (no visible K8s pod/GPU).
• Reuse over re-ingest: always Step 2 first — 57 models + 56 datasets are already there.
• Cleanup is irreversible (DELETE .../{id} + S3 global/models/ scoped delete) — see [[destructive-batch-guard]].
• Do NOT hand-download 40GB+ as a fallback unless the catalog ingest is truly unavailable.
• ⚠️ Large-model ingest can FAIL on slow egress (observed 2026-07-23): Qwen/Qwen-Image-2512 (~40GB) sat at ingest_status: downloading for ~4h then the catalog entry vanished (GET→404, count back to baseline) — the ingest hit a duration/size ceiling over the cluster→HF ~3MB/s egress and was cleaned up. The path is proven for the built-in catalog (mostly LLMs, likely staged from a faster mirror), but a 40GB model over public-HF egress may not finish. Mitigations: verify it reaches available (don't assume); prefer a quantized/smaller variant; or fix the cluster's HF egress. Watch for the entry disappearing = failure, not "still downloading forever."