remote-compute-ssh

You've decided to run this on the user's SSH host. This skill covers the orchestration layer — partitions, env activation, job scripts, file transfer, recovery — not the science; what to run and why comes from the task and its own skills. Each `c.submit_job()` puts an approval modal in front of the user and, once approved, spends their allocation; a string of failed submits costs their attention, their compute, and their trust. So the shape of a good run is: read what's already known about this host, ask once for what isn't, land the first submit, and write down what you learned about the host or compute provider so the next session goes straight to the job. ## Workflow Every `host.compute.*` call in this skill runs via the **`repl` tool** (the control-plane kernel), not the `python` tool. Job submission opens the user's approval modal and the SSH connection from the orchestrator's own process; that has to happen outside the sandboxed data workspace, so `host.compute` simply isn't attached in the `python` tool (you'd see `host has no method 'compute'`). The two kernels share your workspace directory but not memory, so the rhythm is: prepare inputs in a `python` cell (write `./in.dat`, pickle what the job needs), run `create → submit_job` in a `repl` cell and let the cell return — the kernel never blocks on compute. Then call the `wait_for_notification` brain-tool to park until the daemon's poller emits the `compute_done` notification, and return to the `python` tool to read the harvested `hpc/<jobId>/` files. The `repl` tool is stdlib-only (`python -I -S`) — keep pandas/numpy work in the `python` tool and pass data through files. Start with the `compute_details({provider, mode:'read'})` tool, then bind once: `c = host.compute.create(provider)`. The doc's shape tells you how much discovery is left: `### env:` blocks and gotchas mean prior sessions did the legwork — trust it. A bare `## Resources` header means first contact — spend one batched `c.call_command('id; module avail 2>&1 | head -40; ls -la ~', intent=..., login_shell=True)` and one `ask_about_compute` now, before any submit. The header's `scheduler:` line is detection, not ground truth; `none` on a thin login node means a heavy direct-exec job would crowd other users, so when the resources look thin and the details doc has no prior note, ask first. If the prose doc has a known-working activation, write it directly into your `command` (e.g. `source <path>/activate && <tool> ...`). If it doesn't, find one via `c.call_command()` (`module avail X`, `conda env list`, likely app dirs) or ask. Install only once you've established the tool genuinely isn't there — user-space (venv/conda under scratch), via `c.call_command()` for a quick install or as its own `c.submit_job()` if it needs a build node. Whichever route produced an activation, run the entrypoint once via `c.call_command()` before building the real job on it. Then `job = c.submit_job(...)` (see below). `inputs=[{src:'file', dst_filename: ...}]` stages the file for you — there's no `c.upload()` step, and once submitted there's nothing to verify with `c.call_command('cat ...')`; the job reads `./<dst_filename>` from its own workdir. End the cell — `submit_job` returns immediately and the daemon's background poller polls the remote, harvests everything the job wrote into your workspace under `hpc/<jobId>/`, and emits a `compute_done` notification when done. Park on the `wait_for_notification` brain-tool until that notification arrives. Its payload carries `{job_id, status, exit_code, featured_files, output_file_count, ...}` — `featured_files` is the subset matching your featured `outputs:` globs (omitting `outputs:` features everything). Publish what you want with `save_artifacts(payload['featured_files'])` — that step is what gives them provenance and surfaces them in the artifact panel. If you need the full result dict (all `output_files`, `left_on_remote`, etc.), re-enter a `repl` cell and call `r = c.attach_job(job_id).result()` — a non-blocking read of what the poller already harvested. `open(r['output_files'][i])` reads any harvested file directly. Chain a remote-resident output via `inputs:[{remote_path: r['left_on_remote'][i]['uri']}]`. Between the notification and `close()` you can still `c.download(f"{job.workdir}/<file>")` for anything the harvest missed. `c.download('/any/absolute/host/path')` works for **any readable file on the host**, not just job outputs — paths outside scratch/data_roots raise an approval card the user clicks Allow on. When the user asks you to fetch a host file, call `c.download()` with the path they gave; the approval card is the authorization gate, so don't refuse on their behalf and don't `cp` into scratch first to dodge it. Dotfiles / paths under a dot-directory (`~/.ssh/*`, `.gitconfig`, `.env`, …) get a hardened per-file confirmation. `c.close()` once you've confirmed — it cleans up the job workdirs on the host. Hand back the result verbatim. ## What to record The `compute_details()` tool is the only state that survives across sessions, and three of your inputs are the user teaching you how their host works: an `ask_about_compute` answer, a `User: <text>` redirect from a declined approval (they clicked Respond and typed what to do instead), or guidance relayed in the conversation. When one arrives, treat it as a teach loop — read the durable fact, append it via the `compute_details({mode:'append'})` tool with a `per user <date>` tag, echo back what you understood in your next `intent` so the user sees the teaching landed, then act on it. Record an activation/partition/account combination you watched succeed too, tagged with how you know: `verified <date>` if you ran the entrypoint and saw exit 0, `per user` if from `ask_about_compute`, `untested` if inferred. A single inline gotcha ("this tool needs `module load cuda/<ver>` here") is worth keeping; per-job state and transient errors aren't. When asking, ask once per gap and batch related questions ("Which partition and account for GPU jobs, and how do I activate `<tool>`?"). Never ask what one `c.call_command()` would tell you — `module avail` first, then ask for what only the user knows: their account string, which env they prefer, whether you may install. The test for whether something belongs here is whether it is true of the host or compute provider, or true of the work you ran on it. A preemption limit is about the provider; a method choice or a result is about the project, and it will sit in front of every future session on this machine — including unrelated projects — long after it has stopped being true. The same goes for what you learn about the user: that belongs in memory, where it is scoped and correctable. When a session ends and nothing new about the provider came up, the right amount to write is nothing. ## When the job fails Read `r['exit_code']` and the harvested log. An infrastructure failure (wrong partition, env not activated, missing module, OOM, walltime) is yours to fix — adjust `command`, record the fix, fresh `c.submit_job()`. A tool failure (the science tool ran but errored on inputs) may be a bad flag or bad input data; one `c.call_command()` to inspect the log usually says which. Infrastructure-fix retries are cheap on a short smoke test and expensive on a long allocation, so after two failed submits on the same job, ask before a third. If a tool returns `retry_after_user_action: true`, the host itself is unreachable (key not loaded, VPN, host down) — call `ask_about_compute` with the error text and wait; don't loop on your own. ## `c.submit_job()` on SSH `command` is a job script. The host hoists scheduler directives from the top into the dispatch wrapper, so write them as if you were handing the file to `sbatch`/`qsub` yourself — one directive per line starting with the scheduler prefix and a space. The host adds `--job-name`/`--output` bookkeeping (yours can't override those); GPU/time/partition/account are yours. Don't write `--array`/`--chdir`/`--wrap` — submit one job per task instead. PBS (`#PBS -l ...`) and LSF (`#BSUB ...`) follow the same pattern with their prefix; for `scheduler: none`, omit directives entirely. The job runs under a login shell, so tools on the host's default `module`/`conda` PATH are visible — but writing the activation into `command` is still the reliable path (deterministic, and what gets recorded in the details doc). The script runs under `bash -eo pipefail`. If you background subprocesses, `wait` alone returns 0 regardless of their exit codes — capture each pid and `wait $pid` (or `wait -n` in a loop) so a failing branch surfaces as a non-zero `exit_code`. `job.cancel()` sends SIGTERM to the process group; a child that ignores TERM or re-`setsid`s won't be reached, so don't daemonize inside the script. cwd is a fresh per-job workdir under scratch — inputs stage flat there as `./<dst_filename>`. `dst_filename` is a bare filename (no `/` — rejected at submit). Only files under that workdir are harvested; if your tool takes an `--output-dir`, point it at `./out` or `.`, not an absolute path under your home or scratch — anything outside the workdir isn't auto-harvested (pull it afterward with `c.download('/abs/path')`; see the Workflow section for how the approval gate works). If the tool insists on a subdir, end the script with a flatten-to-root step (`cp ./out/*.<ext> ./ 2>/dev/null || true`) so your `outputs:` globs match, plus an `ls -lh` of the expected files so the log shows what's there before harvest. The `|| true` matters: under `-eo pipefail` a missing optional output would otherwise fail the job. `intent` is the approval-modal headline, the one line the user reads to decide whether to let this run on their allocation: name the tool, the target, and the scale; on a retry, say what's different. `inputs` with `{src}` (workspace-relative path or the literal `{{artifact:ID}}` marker — not a kernel-resolved `/sessions/...` path) are staged from this machine; with `{remote_path}` (absolute, under a `data_roots:` entry or scratch) they're symlinked, no transfer. Anything over ~100 MB that already lives on the host should be a `{remote_path}`, not a `{src}` — staging is link-rate and copies into the job workdir. `outputs` — bare string is a featured deliverable; `{glob, visibility:'hidden'}` is diagnostic; `{glob, residency:'remote'}` stays on the cluster and comes back as `left_on_remote`. `harvest:{exclude:['work/**'], max_file_mb, max_total_mb}` caps what the poller pulls. Harvest likewise caps at ~100 MB per file: larger outputs stay on the cluster by default and come back in `left_on_remote` with `reason:'threshold'` — set `residency:'remote'` to choose that, or `max_file_mb`/`max_total_mb` to tighten it. A `left_on_remote` URI is for chaining (`inputs:[{remote_path: uri}]`) or peeking (`c.call_command(f'head -c 4096 {uri_path}', intent=...)`); `c.download()` it only when you or the user actually need the bytes locally — it's link-rate-slow and the file is already where the next job needs it. ```python # repl tool — host.compute isn't attached in the `python` tool c = host.compute.create('ssh:<cluster>') job = c.submit_job( intent='<tool> on <input> — 1 GPU, ~10 min', command='''#SBATCH --gres=gpu:1 #SBATCH --time=15 #SBATCH --partition=<partition> module load <tool>/<ver> <tool> ./in.dat --out ./out cp ./out/*.result ./out/*.json ./ 2>/dev/null || true ls -lh ./*.result ./*.json''', inputs=[ {'src': 'in.dat', 'dst_filename': 'in.dat'}, # workspace-relative (prepared in a `python` cell) {'src': '{{artifact:<id>}}', 'dst_filename': 'ref.dat'}, # artifact marker — either form works # or chain a prior job's harvest: {'src': prev_featured[0], 'dst_filename': 'prev.out'} ], outputs=[ '*.result', # featured {'glob': '*.json', 'visibility': 'featured'}, {'glob': '*.log', 'visibility': 'hidden'}, ], timeout_seconds=900, ) print(job.job_id) # cell ends here — kernel never blocks on compute ``` Then call the `wait_for_notification` brain-tool. The `compute_done` notification payload carries `{job_id, status, exit_code, featured_files, output_file_count, ...}`; act on it directly: ```python # after wait_for_notification returns the compute_done payload — # featured_files paths are workspace-relative under hpc/<jobId>/ save_artifacts(payload['featured_files']) # publish with provenance ``` If you need the fuller result dict (`output_files`, `left_on_remote`, `remote_workdir`, `stdout_tail`): ```python # repl tool — non-blocking read of what the poller already harvested r = c.attach_job(job_id).result() # r → {status, exit_code, output_files, featured_files, left_on_remote, # remote_workdir, ...} c.close() ``` `output_files` is the complete list (uncapped), ordered featured-first; the same files are on disk at `hpc/<job_id>/`. Each `.submit_job()`/`.call_command()` that isn't Always-Allowed shows one approval modal; max 10 — batch fan-out into one job script, or have the user click Always-Allow if you're looping. ## When the user gives you a budget A user who says "stay under twenty nodes" or "keep it to a hundred at a time" is giving you a number that the prompt alone can't enforce. You'll write it into the orchestrator's instructions, but the sub-agents you delegate to start with fresh context — they never see that line, and each one will reasonably try to use as much compute as its own task seems to warrant. Across a wide fan-out that drifts well past whatever the user had in mind, and the first sign is usually the cluster admin's email. `host.compute.set_concurrency_limit(k)` exists so the user's number becomes a property of the session rather than a sentence in a prompt. Call it once before delegating; the daemon stores it against the session root, counts every sub-agent's live job against the same `k`, and quietly holds any submit that would put the session over (the SDK retries with backoff under the hood). Sub-agent code is unchanged — the hold sits below `submit_job`, not in the agent. Choosing `k` has one constraint beyond the user's intent: each provider also has its own ceiling, and that ceiling refuses rather than queues. A session limit above it doesn't fail, it just stops being the binding constraint — submits past the host's own ceiling error instead of waiting. `host.compute.status()` returns both your `k` and the provider ceilings, so you can pick a value that actually queues. When the user hasn't given a number, leaving the limit unset keeps today's behaviour; set one yourself only if a fan-out is wide enough to threaten the host cap and you'd rather queue than fail. ## Submitting several jobs Submitting a batch and harvesting them as each finishes uses the same `wait_for_notification` mechanism, just called repeatedly. The poller tracks every job you submitted, harvests each independently when the remote reports it terminal, and posts one `compute_done` per job; each `wait_for_notification` call returns whatever's queued (one or more) and then blocks for the next. ```python # repl tool — submit, print ids, end the cell c = host.compute.create("ssh:gpu-cluster") jobs = [ c.submit_job( command=f"python fold.py --seed {s} --in input.fasta --out ranked.pdb", intent=f"AlphaFold seed {s}", inputs=[{"src": "input.fasta", "dst_filename": "input.fasta"}], outputs=[{"glob": "*.pdb", "visibility": "featured"}], timeout_seconds=3600, ) for s in range(5) ] print({j.job_id: j.status for j in jobs}) ``` Then loop the brain tool. Each call's `notifications` list may contain more than one entry if two jobs finished while you were processing the previous batch, so iterate it; the loop ends when the call returns `{status:'error'}` because no compute jobs remain. ```text wait_for_notification(timeout_seconds=1800) → {status:'received', notifications:[ {notification_type:'compute_done', payload:{job_id:'…', intent:'AlphaFold seed 3', status:'success',