b200-exp-host

name

b200-exp-host

description

Connect to and operate the hyojung-2gpu B200 experiment host (2x NVIDIA B200, 10.7.60.194) over SSH for GPU inference/serving experiments. Use when the user says "B200 접속", "B200 서버에서", "hyojung-2gpu", "실험 서버 접속", "B200 호스트에 명령", "ssh into the B200 box", "run on the B200 host", "GPU 서버에서 vllm 돌려", or wants to run/inspect anything on that bare-metal 2-GPU box. Provides a thin SSH wrapper (scripts/b200.sh) reading credentials from a repo-EXTERNAL env file. Do NOT use for the H200 K8s demo cluster (use h200-gpu-usage-inspector), the SKT B200 fleet lifecycle (use gpu-lifecycle-manager), or GPU hardware health grading (use gpu-health-report). For the vLLM P/D disaggregation experiment WORKFLOW that runs on this host, use vllm-pd-disagg-benchmark (it calls this skill to connect).

B200 Experiment Host Access

Thin, credential-safe SSH access to the hyojung-2gpu bare-metal box: 2× NVIDIA B200 (183 GB each), GPU0↔GPU1 NVLink (NV18), driver 580.95.05. This skill is the connection layer — the experiment logic lives in vllm-pd-disagg-benchmark.

Step 1: Detect credentials + tooling (always first)

The host password is NEVER stored in this repo. It lives in a repo-external file.

ENV_FILE="${B200_ENV:-$HOME/.config/b200-exp/host.env}"
test -f "$ENV_FILE" && echo "creds ok" || echo "MISSING: create $ENV_FILE (0600)"
command -v sshpass >/dev/null && echo "sshpass ok" || echo "install: brew install hudochenkov/sshpass/sshpass"

Decision tree:

• creds file missing → create it (see "Credentials" below), then continue.
• sshpass missing → brew install hudochenkov/sshpass/sshpass.
• both present → use scripts/b200.sh for everything (do not hand-roll ssh).

Step 2: Connect and probe

Always verify the host before running work. The wrapper resolves creds itself.

bash .claude/skills/b200-exp-host/scripts/b200.sh check

Returns hostname, per-GPU memory/util/driver, NVLink topology, vllm/venv readiness, and disk. If check fails on auth → password rotated, update the env file. If it fails on connect → host down/unreachable (report; do not retry >2×).

Step 3: Run work on the host

Pick the method by duration (this is the decision that matters):

Need	Command	Why
Short command (<~1 min)	b200.sh run 'nvidia-smi'	streams output back
Long job (install, download, serve, bench)	run via the Bash tool with run_in_background: true wrapping b200.sh run '...'	harness re-invokes you on completion; SSH stays attached
Fire-and-forget on host	b200.sh runbg 'CMD'	setsid nohup, survives SSH close, returns remote log path to poll
Copy file up / down	b200.sh put LOCAL REMOTE / b200.sh get REMOTE LOCAL	scp with same creds
Quick GPU snapshot	b200.sh gpu	one-line mem/util per GPU

Prefer the Bash tool's run_in_background for installs/downloads/servers over remote nohup polling — you get an automatic completion signal and clean logs.

Step 4: Report

State what ran, on which GPU(s), and the result. For GPU-occupancy questions, report per-GPU memory.used/total and utilization.gpu from b200.sh gpu.

Credentials (repo-external, one-time setup)

Create ~/.config/b200-exp/host.env (chmod 600), NOT in the repo:

export B200_HOST=10.7.60.194
export B200_USER=root
export B200_PORT=22
export B200_SSHPASS=<password>       # ask the user; never commit
export B200_WORKDIR=/root/vllm-pd-exp
export B200_VENV=/root/vllm-pd-exp/.venv

Auth is password-based (sshpass). To switch to keys later, add your pubkey to the host's /root/.ssh/authorized_keys and drop B200_SSHPASS.

gotchas (from production use)

• Password auth only so far — the box rejects local keys (Permission denied (publickey,password)) until a pubkey is authorized. sshpass is required.
• No CUDA toolkit in PATH (nvcc absent) — that's fine; vLLM's pip wheel bundles the CUDA runtime (torch cu130). Don't try to "fix" missing nvcc.
• Driver 580.95.05 + CUDA-13 torch — Blackwell (sm_100) works; don't downgrade.
• HF downloads are unauthenticated (no token on host) → rate-limited but public Apache-2.0 models still complete. Set HF_TOKEN on the host to speed up.
• Large models AND datasets: use the internal registry, NOT HuggingFace ([[gpu-model-registry]]). Cluster/host egress is slow (~4MB/s); a 66GB model = hours. Pre-staged copies live at s3://ai-platform/global/{models,datasets}/<ID> (SeaweedFS ~180MB/s). Pull with scripts/skills/model_registry.py [--kind datasets] ls|pull (stdlib, no pip), or add new ones via metis-catalog-ingest / model_registry.py push.
• Canonical venv = /root/vllm-pd-exp/.venv; canonical workdir /root/vllm-pd-exp. Other people's caches live in /data/cache_dir/huggingface (shared, read-only-ish).
• Never commit the password. It is only in ~/.config/b200-exp/host.env (0600).
• Don't paste the password into b200.sh run strings — the wrapper injects it via the SSHPASS env, keeping it out of the remote process list where possible.

Reference

• scripts/b200.sh — the SSH/scp wrapper (check/run/runbg/put/get/gpu).
• Experiment workflow that uses this host: vllm-pd-disagg-benchmark.