serve-model-vibe-test

name	serve-model-vibe-test
description	Stand up a PUBLIC, shareable inference endpoint for an HF/gs model on an Iris TPU so people on the internet can vibe-test it in a browser or via an OpenAI-compatible API. Use when asked to "serve a model for testing", "throw up a public endpoint", "let people play with model X", or to demo a checkpoint. Combines marin-serve (marin#6556) + a Pinggy tunnel from our endpoint bank.

serve-model-vibe-test

Spin up a public endpoint for a HuggingFace repo or gs:// checkpoint. marin-serve runs vLLM on a single-host Iris TPU slice behind the controller proxy; a Pinggy tunnel makes its proxied port public at https://<id>.a.pinggy.link/proxy/serve.<ep>/, with a dashboard and OpenAI-compatible API.

scripts/inference/serve_public.py wraps marin-serve, hpc.pinggy_utils.PinggyTunnel, and the Pinggy bank.

⚠️ Security — read first

The public URL is unauthenticated. Use it only for throwaway vibe testing; keep --timeout-hours short, use an unused Pinggy pair, and tear it down when done.

Prerequisites

• marin-serve available (marin#6556). serve_public.py invokes ~/Documents/marin/.venv/bin/python -m marin.inference.quick_serve_cli by default. Override with --marin-serve-bin or MARIN_SERVE_BIN.
• Run from the marin repo root (~/Documents/marin), not from this repo. marin-serve bundles Path.cwd() as the worker workspace (quick_serve_cli.py:227) and the worker runs uv sync --all-packages --extra tpu --extra vllm against it. Only the marin workspace defines the tpu/vllm extras (on marin-core); any other CWD → Extra 'tpu' is not defined in any project's 'optional-dependencies' table and the job dies before vLLM starts. serve_public.py resolves its own repo root via parents[2] for the hpc import, so invoke it by absolute path from the marin CWD.
• Iris controller access (the laptop's step-ca/GCP creds — same as iris CLI).
• $DC_AGENT_SECRET_ENV sourced (Pinggy uses your ~/.ssh identity; SSH to pro.pinggy.io:443).
• Pinggy bank at ~/Documents/notes/ot-agent/pinggy_bank.md (override --pinggy-bank / PINGGY_BANK).
• Single-host TPU slices only (v6e-8, v6e-4, v5litepod-8, …); multi-host is rejected.

Pick an unused Pinggy pair

The bank has 10 pairs. Before launching, pick one that isn't already serving — a quick probe (a free pair returns a Pinggy "tunnel not found"/503 or connection error; an in-use one returns your model):

for i in 1 2 3; do u=$(sed -n "/## Pair $i$/,/pinggy.link/p" ~/Documents/notes/ot-agent/pinggy_bank.md | grep pinggy.link); \
  echo "pair $i: $u -> $(curl -s -o /dev/null -w '%{http_code}' --max-time 6 https://$u/ || echo down)"; done

Use a pair that shows down/502/503 (free). Pass it as --pair N.

Launch (the one-liner)

cd ~/Documents/marin                               # MUST be the marin repo — see Prerequisites
set -a; source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"; set +a
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py <MODEL> --tpu <SLICE> [--region <R>] \
    [--chat-template <FILE-or-URL>] --pair <N> --timeout-hours 6

It launches marin-serve, waits for READY — dashboard: http://127.0.0.1:<port>/proxy/serve.<ep>/, opens the Pinggy tunnel, and prints:

  dashboard : https://<id>.a.pinggy.link/proxy/serve.<ep>/
  OpenAI    : https://<id>.a.pinggy.link/proxy/serve.<ep>/v1

Keep the process running because it holds both tunnels. For unattended operation, use tmux, not nohup setsid … on macOS:

tmux new-session -d -s serve-public \
  "source \"${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}\" && cd ~/Documents/marin && \
   python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py <MODEL> ... 2>&1 | tee /tmp/serve_public.log"
# then poll /tmp/serve_public.log (or `tmux attach -t serve-public`) for the "OpenAI :" line

Worked example — the Delphi 9.7B SFT canary (marin#6545)

This is the reference model: laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft. It needs its own chat template (the repo ships a plain Llama-3 one); use delphi_v0.jinja2. marin-serve auto-derives the 4k context (malformed RoPE) and TP=2 (30 heads on a 4-chip slice).

# run from ~/Documents/marin (the bundled workspace); script is invoked by absolute path
cd ~/Documents/marin && source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}"
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py \
    laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft \
    --tpu v6e-4 --region europe-west4 \
    --chat-template https://raw.githubusercontent.com/open-thoughts/OpenThoughts-Agent/ed4d6f483151f14d6d78cf732f04cd3c8ff5c606/chat_templates/delphi_v0.jinja2 \
    --pair 1 --timeout-hours 6

Verify once the public URL prints:

BASE=https://<id>.a.pinggy.link/proxy/serve.<ep>
# from the laptop you must force the real Pinggy edge IP (ISP DNS poisons *.a.pinggy.link); see Gotchas
curl "$BASE/v1/models"
curl "$BASE/v1/chat/completions" -H 'content-type: application/json' \
  -d '{"model":"laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft",
       "messages":[{"role":"user","content":"Give me a fun fact about otters."}]}'

(Base/midtrained checkpoints with no chat template auto-default the dashboard to completion mode — omit --chat-template.)

Teardown

• Foreground: Ctrl-C (tears down the Pinggy tunnel and the marin-serve job it launched).
• Detached/other host: iris job stop <job> --cluster marin (the job name is in the log / iris query), then kill the serve_public.py PID. The slice also self-stops at --timeout-hours.

Gotchas

• Run from the marin repo or the build fails before vLLM starts. Path.cwd() becomes the worker workspace; only marin defines the needed extras. Invoke serve_public.py by absolute path from ~/Documents/marin.
• Dead tunnel reported live (ps state T): run kill -CONT <ssh_pid> <loop_pid>, then re-probe. PinggyTunnel.start() prevents this with detached stdin, ssh -n, and setsid; serve_public.py prints LIVE ✓ only after a DNS-aware /v1/models HTTP 200 probe.
• Cold compile: first boot of a model can take tens of minutes; --ready-timeout (default 2700s) bounds the wait.
• Don't use marin-serve --no-wait for the public path — the local proxied port only exists while the launching process runs; --no-wait returns immediately and there's nothing to tunnel.
• Pinggy pair collisions: two jobs on the same pair clobber each other — always pick a free pair.
• Single-host concurrency: one slice = limited throughput; fine for a few testers, not a viral launch (that's inference-broker territory).
• The public URL carries the /proxy/serve.<ep>/ base path (it's the controller proxy path), so the OpenAI base is https://<id>.a.pinggy.link/proxy/serve.<ep>/v1, not /v1.
• Local DNS poisoning: verify through the real edge with dig @1.1.1.1 and curl --resolve, or use 1.1.1.1/8.8.8.8 DNS. See .agents/projects/pinggy/pinggy.md.