nvcf-self-managed-cli

<!-- Token Budget: - Level 1 (YAML): ~120 tokens - Level 2 (this file): ~1900 tokens (target <2000) - Level 3 (prompts/, reference/, examples/): loaded on demand --> # NVCF Self-Hosted CLI `nvcf-cli` drives every step of bringing up self-hosted NVIDIA Cloud Functions: cluster registration, control-plane install, compute-plane install, function deploy/invoke, and lifecycle management. Use this skill any time the user wants to operate self-hosted NVCF. ## When to use - "install self-hosted NVCF" / "bring up an NVCF cluster" - "register a (compute|GPU) cluster with NVCF" - "deploy a (container|GPU) function" / "invoke an NVCF function" - "check NVCF cluster health" / "is my NVCF install OK?" - "rotate NVCF cluster JWKS" / "the NVCA agent stopped authenticating" - "tear down NVCF" / "remove the compute plane" / "uninstall NVCF" / "deregister this cluster" - "preview what `down` would do" / "dry-run uninstall" - Any task operation: create / run / submit / monitor / cancel / delete / list tasks, helm task, update task secrets, retrieve task results, bulk fetch task details. - Any reference to `NVCT`, `NVCT task`, or `NVCT batch job` (list / run / cancel / delete / results). - Any reference to `NVCFBackend`, `NVCA`, ICMS, helm releases like `helm-nvcf-*`, or `icms.<domain>` / `api.<domain>` URLs. ## Quick start For remote one-click installs, prepare Gateway API ingress and CLI endpoint configuration before running `self-hosted up`. The command applies the control plane and then immediately calls API, API Keys, invocation, and gRPC endpoints. If the Gateway is not programmed or the CLI host headers do not match the HTTPRoutes rendered by the stack environment, post-install health and cluster registration will fail. ```sh # Single-cluster (control + compute on the current kubeconfig context): nvcf-cli self-hosted up --cluster-name=ncp-local # Split-cluster (control plane on context A, compute plane on context B): KUBECONFIG=cp.yaml:gpu1.yaml nvcf-cli self-hosted up \ --cluster-name=ncp-local \ --control-plane-context=admin@cp \ --compute-plane-context=admin@gpu1 \ --icms-url=https://icms.nvcf.example.com # Add a new compute plane to an existing control plane (no kubectl access to CP needed; # reaches the control plane via the public ICMS HTTPRoute): nvcf-cli self-hosted add-compute-plane \ --cluster-name=ncp-local-2 \ --compute-plane-context=admin@gpu2 \ --icms-url=https://icms.nvcf.example.com \ --token=$ADMIN_JWT # Tear down (always plan-only first): nvcf-cli self-hosted down --plan-only --cluster-name=ncp-local --json | jq nvcf-cli self-hosted down --cluster-name=ncp-local # Per-plane uninstall (GitOps; mirrors `install`): nvcf-cli self-hosted uninstall --no-apply --compute-plane --cluster-name=ncp-local | kubectl delete -f - ``` > **`up` vs `add-compute-plane`.** `up` always installs both planes — use it for the *first* install. `add-compute-plane` is the right subcommand any time the control plane is already running and you want to attach an Nth compute cluster. > > **`down` always with `--plan-only` first.** Show the user the `willUninstall.commands[]` array before running for real. ## Authentication `nvcf-cli` uses two credentials. Each is resolved independently in this order: | Precedence | Source | |---|---| | 1 (highest) | Environment variable (`NVCF_TOKEN`, `NVCF_API_KEY`) | | 2 | Config file (`~/.nvcf-cli.yaml`, keys `token` and `api_key`) | | 3 | State file (`~/.nvcf-cli.state`), skipped when the stored token is expired | | Env var | Token type | Used by | |---|---|---| | `NVCF_TOKEN` | Admin JWT | `function create` / `deploy` / `update` / `delete`, cluster management (`cluster register`/`list-registered`/`rotate`/`delete`, `self-hosted` ops). Required for admin commands; preferred for the rest. | | `NVCF_API_KEY` | `nvapi-...` API key | `function invoke` / `list` / `get`, queue details. Falls back to `NVCF_TOKEN` when unset. | Token generation flow: 1. Point the CLI at your self-hosted endpoints. Defaults target production NVCF; override with env vars or a yaml file (`--config <path>`, `./.nvcf-cli.yaml`, or `~/.nvcf-cli.yaml`; a full template ships at `src/clis/nvcf-cli/.nvcf-cli.yaml.template`). `NVCF_BASE_HTTP_URL` / `API_KEYS_SERVICE_URL` / `NVCF_BASE_GRPC_URL` tell the CLI *where to send requests*. `API_HOST` / `API_KEYS_HOST` / `INVOKE_HOST` set the `Host:` header on those requests — needed when every NVCF service sits behind a single gateway that routes by hostname (Envoy Gateway HTTPRoute on `api.<ELB>`, `api-keys.<ELB>`, `invocation.<ELB>`, `llm.<ELB>`). Skip the three Host vars when the base URL already resolves to the right service. Env-var form: ```sh export NVCF_BASE_HTTP_URL=https://gw.example.com export API_KEYS_SERVICE_URL=https://gw.example.com export NVCF_BASE_GRPC_URL=gw.example.com:443 # Only when the gateway routes by Host header: export API_KEYS_HOST=api-keys.gw.example.com export API_HOST=api.gw.example.com export INVOKE_HOST=invocation.gw.example.com ``` Yaml form (`~/.nvcf-cli.yaml`): ```yaml base_http_url: https://gw.example.com api_keys_service_url: https://gw.example.com base_grpc_url: gw.example.com:443 # Only when the gateway routes by Host header: api_keys_host: api-keys.gw.example.com api_host: api.gw.example.com invoke_host: invocation.gw.example.com ``` 2. `nvcf-cli init` calls the API Keys service via those endpoints and writes the admin token to `~/.nvcf-cli.state`. Use `nvcf-cli refresh` to rotate a stored token. 3. After `init`, `nvcf-cli api-key generate` mints function + task API keys (also stored in state). After `init`, the credentials live in `~/.nvcf-cli.state`, so later commands work without exporting `NVCF_TOKEN` or `NVCF_API_KEY`. Export the env var only to override the stored value (for example a shorter-lived token from CI). ## Core subcommands | Subcommand | What it does | When to use | |---|---|---| | `nvcf-cli self-hosted check --pre [--local-only \| --control-plane-context=X \| --compute-plane-context=Y]` | Pre-flight: local-host tools + cluster-side prerequisites | Always run first on a new environment | | `nvcf-cli self-hosted install --control-plane \| kubectl apply -f -` | Render + apply the control plane | When you want manual control over apply (GitOps-friendly) | | `nvcf-cli self-hosted install --compute-plane --cluster-name=X \| kubectl apply -f -` | Register cluster + render compute plane | Same — manual apply path | | `nvcf-cli self-hosted up --cluster-name=X` | One-shot first install: pre-flight → control plane → register → compute plane | Standard install path (both planes from scratch) | | `nvcf-cli self-hosted up --plan-only --cluster-name=X` | Dry-run: emit phase-by-phase plan + ETA without changing state | Agent / CI preview before commit | | `nvcf-cli self-hosted add-compute-plane --cluster-name=X --compute-plane-context=Y --icms-url=… --token=$JWT` | Add a new compute plane to an *existing* control plane (no CP install) | Adding the 2nd, 3rd, … GPU cluster after the initial `up` | | `nvcf-cli self-hosted uninstall --control-plane` | Per-plane primitive: `helmfile destroy` on the control plane (refuses if compute planes still registered) | Final teardown after all compute planes are gone, or scripted pipelines | | `nvcf-cli self-hosted uninstall --compute-plane --cluster-name=X` | Per-plane primitive: `helmfile destroy` on a compute plane (no ICMS unregister, no drain) | Just remove the helm releases without the ICMS-side cleanup | | `nvcf-cli self-hosted uninstall --no-apply <plane>` | Render delete YAML via `helm get manifest` | GitOps; `\| kubectl delete -f -` or commit + Argo applies | | `nvcf-cli self-hosted down --cluster-name=X` | Orchestrator: drain → uninstall --compute-plane → cluster delete in ICMS | Standard "remove one GPU cluster" path | | `nvcf-cli self-hosted down --all --confirm` | Orchestrator: tear down every registered compute plane + control plane | Full uninstall | | `nvcf-cli self-hosted down --plan-only --cluster-name=X` | Dry-run preview (phases + helm releases + ICMS rows + ETAs; no helm/helmfile contact) | **ALWAYS run before any actual `down`** | | `nvcf-cli self-hosted status [--cluster-name=X] [--watch] [--json]` | Snapshot dashboard of cluster identity + component health + recent events | Routine health checks; `--watch` for live | | `nvcf-cli init` | Mint admin token from API Keys service via the public api gateway | Before any cluster-management operation; idempotent | | `nvcf-cli cluster register --name=X --nca-id=Y --region=Z [--ignore-existing]` | Register a cluster JWKS+OIDC issuer with ICMS | Standalone register (without compute-plane install) | | `nvcf-cli cluster list-registered --nca-id=Y [--icms-url=URL]` | List self-hosted cluster registrations from ICMS | Check registered compute-plane names and IDs with the admin token | | `nvcf-cli cluster rotate --cluster-id=ID` | Rotate cluster JWKS in ICMS | When NVCA's K8s signing key changed and PSAT verification started 401-ing | | `nvcf-cli cluster delete --cluster-id=ID` | Remove cluster registration from ICMS | **Confirm with user.** Destroys ICMS state for the cluster. | | `nvcf-cli api-key generate --description="…" --expires-in=1h` | Mint both a function API key and a task API key (default) | Before invoking functions or creating tasks; run after every `init` | | `nvcf-cli api-key generate --for function --description="…"` | Mint a function API key only (`invoke_function` scope) | When only function invocation is needed | | `nvcf-cli api-key generate --for task --description="…"` | Mint a task API key only | When only task operations are needed | | `nvcf-cli function create --input-file=<json>` | Create function metadata in ICMS | First step of any function deploy; use `functionType: "LLM"` and `models[].llmConfig` for LLM functions | | `nvcf-cli function deploy create --input-file=<json>` | Schedule a deployment of a created function | Waits for ACTIVE before returning (timeout 900s) | | `nvcf-cli function invoke --input-file=<json>` | Invoke a deployed function | Requires API key (not admin token) | | `nvcf-cli function delete --function-id=ID --version-id=VID` | Remove a function and its deployment | **Confirm with user.** | | `nvcf-cli task create --name=X --gpu=H100 --instance-type=Y --image=Z` | Submit a container task; saves task ID to CLI state | Requires task API key; set `NVCF_BASE_NVCT_URL` to the NVCT gateway endpoint | | `nvcf-cli task create --name=X --gpu=H100 --instance-type=Y --helm-chart=Z` | Submit a Helm task | Same as container task; `--helm-chart` replaces `--image` | | `nvcf-cli task create --input-file=<json>` | Submit a task from a JSON config file | Recommended for repeatable configurations | | `nvcf-cli task list [--status=QUEUED\|RUNNING\|COMPLETED]` | List tasks, optionally filtered by status | | | `nvcf-cli task get [taskId]` | Get details for a task; uses saved task ID if omitted | | | `nvcf-cli task events [taskId]` | Stream lifecycle events for a task | | | `nvcf-cli task results [taskId]` | List result artifacts for a completed task | Result upload not yet supported; returns empty list for `NONE` strategy | | `nvcf-cli task cancel [taskId]` | Cancel a queued or running task | | | `nvcf-cli task delete [taskId]` | Delete a task record | **Confirm with user.** | | `nvcf-cli task update-secrets [taskId] --secrets NAME=value` | Update secrets on a task; supplied secrets are added or updated by name, existing secrets not in the request are preserved | | | `nvcf-cli task bulk --task-ids=ID1,ID2` | Fetch details for multiple tasks in one call | | > Registry credential propagation: after `registry-credential add`/`update`/`delete`, task creation can keep using the previous credential for up to about 5 minutes (NVCT caches account credentials at `nvct.nvcf.cache-ttl`, default `PT5M`), even though `registry-credential list`/`get` show the new value immediately. Wait about 5 minutes, or `kubectl -n nvcf rollout restart deployment/nvct-api` to apply immediately. See [reference/troubleshooting.md](reference/troubleshooting.md). ## LLM function type Use `functionType: "LLM"` for OpenAI-compatible models served through the self-managed LLM Gateway. LLM functions must define at least one `models[]` entry with `name` and `llmConfig.uris`; the supported upstream paths are `/v1/chat/completions`, `/v1/responses`, and `/v1/embeddings`. Use `/health` on port `8000` as the default OpenAI-compatible container health probe unless the image exposes a different readiness path. LLM function type is independent of workload packaging. For a Helm-chart backed LLM function, keep `functionType: "LLM"` and `models[].llmConfig`, then set `helmChart` and `helmChartServiceName` in the create request. `helmChartServiceName` must match the Kubernetes Service exposed by the chart, and `inferencePort` must be that Service port. Invocation uses the LLM route, for example `https://llm.invocation.<domain>/v1/chat/completions`. The OpenAI `model` value must be `<function-id>/<model-name>`; the function ID is the routing key and the model name is forwarded upstream. Update mutable per-model routing settings with `nvcf-cli function update --llm-model-update='name=<model>,routingMethod=<method>,tokenRateLimit=<limit>'`, or put the same fields under `modelUpdates[].llmConfig` in an update JSON file. See [reference/flags.md](reference/flags.md) for accepted routing methods. `tokenRateLimit` supports positive integer limits for `S`, `M`, `H`, `D`, and `W`; use JSON input for combined limits such as `1000-S,5000-M,100000-H,500000-D,1000000-W`. Do not include `uris` in model updates. For `/v1/responses`, the gateway proxies the native Responses path upstream, relays SSE to streaming clients, and aggregates the terminal JSON response for non-streaming clients. For `/v1/embeddings`, input may be a string or string array, must be non-empty, and may contain at most 2048 entries. Session stickiness uses `x-multi-turn-session-id` for chat completions and Responses API requests only. Embeddings requests do not use stickiness. ## Common workflows For step-by-step playbooks, load the prompt that matches the user's intent: - **Install from scratch.** [prompts/install-from-scratch.md](prompts/install-from-scratch.md) — k3d cluster → preflight → up → deploy a smoke function. - **Add a new compute plane.** [prompts/add-compute-plane.md](prompts/add-compute-plane.md) — split-cluster `up` against an existing control plane. - **Deploy and invoke a function.** [prompts/deploy-and-invoke.md](prompts/deploy-and-invoke.md) — create → deploy → API key → invoke, including the LLM create/invoke variant. - **Diagnose a failed install.** [prompts/diagnose-failed-install.md](prompts/diagnose-failed-install.md) — `status --json` → identify failed component → kubectl describe → remediation.