remote-compute-ssh

This skill covers remote compute over SSH, including direct execution and Slurm submission: listing hosts, creating handles, running short remote commands (callCommand), reading/writing host knowledge docs, and the full async job lifecycle — submit → save `job_id` → read non-blocking snapshots by that ID → harvest → analysis turn → publish artifacts. **Where host.compute runs:** `host.compute` lives ONLY on the control-plane REPL kernel — run every example below with the `repl_execute` tool (JavaScript), the same kernel that hosts `host.mcp`. The `python`/`r` data kernels have NO `host.compute` (SSH and approvals stay outside the sandbox workspace); calling it from a python/r cell will fail with `host.compute is undefined`. ## Choose an execution location Only Compute Hosts enabled for this Session are visible or callable. Discover them in one catalog; each entry has role `selected` or `available`. A non-empty selected pool is an execution instruction: run tool-backed task work on one or more selected hosts as the task requires. The pool has no priority and does not imply automatic multi-host scheduling. If no host is selected, choose from the available entries. Read `details()` only for candidates that need closer evaluation. Never guess or reuse a provider id absent from the catalog. A user naming a disabled host does not make it callable; explain that it must first be enabled for this Session. If no eligible host is usable, explain the blocker and ask the user how to proceed. ```javascript const hosts = await host.compute.listHosts() const selectedHosts = hosts.filter((host) => host.role === 'selected') const candidates = selectedHosts.length > 0 ? selectedHosts : hosts ``` Each list item is a compact summary with `provider_id`, `display_name`, `shape`, `execution_mode`, `status`, and `role` (`last_probe_ok`, `probe_failed`, or `not_probed`). `last_probe_ok` means the most recent persisted Probe succeeded; it does not assert live connectivity. Knowledge documents and resource probe snapshots are deliberately excluded from discovery results. ## API reference ```javascript // List this Session's enabled hosts as one role-bearing compact catalog const hosts = await host.compute.listHosts() // Compatibility discovery names remain available; both still hide disabled hosts. const visibleHosts = await host.compute.listRegistered() const selectedHosts = await host.compute.listPreferred() // Create a handle to a specific host (no network call) const c = host.compute.create('ssh:<alias>') // Run a short remote command (throws on approval_denied / host_unreachable / timeout) const result = await c.callCommand('<shell command>', '<one-line intent for the approval card>', { loginShell: true, // default: true — runs login profiles, then readable ~/.bashrc, before this command timeoutSeconds: 60 // optional — the host applies its own default (60s) when omitted }) // result → { exit_code, stdout, stderr, truncated } // Read the persisted operation instructions and the independent resource probe snapshot. // doc is always the exact saved text (including '' before instructions are saved). // probe is explicitly null when this host has never been probed. const info = await host.compute.details('ssh:<alias>', { mode: 'read' }) // Append a note to the persisted host knowledge doc (agent writes; 32 KB cap enforced). // Append changes only doc; it never copies or changes probe observations. await host.compute.details('ssh:<alias>', { mode: 'append', text: '\n## Note\nlearned X on <date>' }) // Alternatively, replace the entire host knowledge doc. Read again before replacing, // especially if you appended above: oldText must match the persisted doc exactly. const latest = await host.compute.details('ssh:<alias>', { mode: 'read' }) await host.compute.details('ssh:<alias>', { mode: 'replace', text: '<new full doc>', oldText: latest.doc }) ``` Treat `doc` as operation instructions and durable host knowledge. Treat `probe` as a dated observation: resource values may change, and detecting a scheduler does not authorize submitting a job or select an account, partition, or queue. On a document mismatch or `details_conflict` error, read again and merge your draft with the latest document before retrying. A resource-only probe refresh does not change `doc` or cause a replacement conflict. After writing, read again to verify the saved contents. A successful append or replace result is only `{ ok: true }`. The write result does not contain `doc` or `probe`; always read again to verify the exact persisted `doc` rather than reading fields from the write result. With `loginShell: true`, the remote Bash login profiles run first and then Open-Science attempts to source `~/.bashrc` when it is readable. A `.bashrc` can deliberately return early for non-interactive shells, so variables declared after such a guard are not available. A missing `.bashrc` is a no-op. Set `loginShell: false` to run the command without either initialization step. Initialization failures are reported through the normal command result/error behavior. ## API reference (async jobs) Use `submitJob` for long-running computations (minutes to hours). It returns immediately with a `job_id`; the job runs on the remote host in the background. When the job finishes, the app automatically harvests the outputs and initiates a new analysis turn if you have not already read the terminal result. Save the exact `job_id` from the submission result in your working context; there is intentionally no historical Job scan for rediscovering it. Status and result reads use only that saved ID and return non-blocking local snapshots. For a local input, `src` is relative to the Agent Session workspace—the same workspace used by file writing tools. Write a script or small generated input there, then pass its relative path. Open Science snapshots accepted inputs before approval and dispatch. Do not pass arbitrary absolute local paths or copy files into app-managed `notebooks/...` directories. An absolute `src` is valid only when it is the exact path returned by `host.artifactPath(versionId)` or an exact registered Session input path already supplied in the Notebook context. ```javascript // Reuse the `candidates` selected above from the Session catalog. // Submit a non-blocking job — returns immediately after the user approves. // The Compute Host's configured execution mode selects direct SSH or Slurm. const c = host.compute.create('ssh:<alias>') const job = await c.submitJob( '<one-line intent for the approval card>', // shown in the approval card '<shell command>', // command to run remotely { environment: 'protein-gpu', // optional logical name; see Environment activation below timeoutSeconds: 3600, // optional; default 24 h, max 7 days inputs: [ { src: 'in.dat', dstFilename: 'in.dat' }, // stage an Agent Session workspace file { remotePath: 'ssh:<alias>/<abs_path>' } // link a remote file (no transfer) ], outputs: [ '*.result', // featured (default visibility) { glob: '*.json', visibility: 'featured' }, // explicitly featured { glob: '*.log', visibility: 'hidden' }, // hidden (diagnostic, not shown in card) { glob: 'checkpoints/**', residency: 'remote' } // leave on remote — recorded in left_on_remote ], harvest: { exclude: ['work/**'], // never harvest these paths maxFileMb: 100, // single-file hard maximum (100 MiB) maxTotalMb: 500 // per-job hard maximum, including stdout/stderr (500 MiB) } } ) // job → { job_id, provider_id, status: 'submitted' | 'queued', remote_workdir } const savedJobId = job.job_id // retain this exact id for the later dependent step return { ...job, job_id: savedJobId } ``` ### Read a saved Job snapshot Use the saved ID when the Job's state or result is relevant. `.status()` and `.result()` are non-blocking local reads in every state; neither waits for completion, triggers SSH, or starts another harvest. `.result()` also includes harvested file lists. Both calls report `follow_up_delivery`. A final `.result()` read returns `suppressed` when it prevents the fallback, or `committed` when that fallback already crossed its dispatch fence. A `.status()` snapshot remains `pending` because it omits harvested file lists. Use the submission's exact ID rather than searching old Jobs. ```javascript const snapshot = await c.attachJob(savedJobId).result() if (!snapshot.result_final) { return { job_id: savedJobId, status: snapshot.status, result_final: false, follow_up_delivery: snapshot.follow_up_delivery } } return snapshot ``` Treat only `result_final: true` as the final result. It means the automatic response is ready: `success`, `failed`, and `timeout` can still be waiting for local harvest, while `cancelled` and `error` are final without waiting for partial output collection. `output_files` contains only files actually available locally; an empty list does not prove there are no remote outputs. The app owns provider polling and harvest in the background. An unread final result is delivered in a later Agent Turn. A final `.result()` snapshot reports `follow_up_delivery: 'suppressed'` when it suppresses that fallback, or `committed` if automatic delivery already won the race and remains authoritative. `.status()` never consumes the full result. ### Direct SSH or Slurm The Compute Host's configured execution mode selects how every job is launched. `direct_ssh` runs the command as a detached process on the SSH target. `slurm` submits it with `sbatch`; put the cluster's required `#SBATCH` directives at the top of `command`. Open-Science owns submission, scheduler-status polling, cancellation, and harvest. Do not call `sbatch`, `squeue`, or `scancel` around `submitJob` yourself. For a Slurm status error or failed cancellation, read [Slurm troubleshooting](references/slurm-troubleshooting.md). In that exceptional path, `compute.callCommand` can inspect the same job and its outputs and perform recovery within the user's existing task authorization. Keep ordinary submission on the Compute API. Read `listHosts()` for the configured mode and `details()` for provider-specific directives; do not try to override the mode per job or infer it only from the workload. If Slurm is unavailable or rejects the script, report the returned error and the concrete next step (for example, add an account or partition directive). Do not silently rerun the workload directly on a login node. Open-Science accepts ordinary single-job directives such as partition, account, CPUs, memory, and GPUs. Set `timeoutSeconds` for the workload runtime. You may set the scheduler allocation limit with one `#SBATCH --time=value` directive; when it is absent, Open-Science derives a default allocation limit from `timeoutSeconds`. Open-Science owns the job name, working directory, stdout, and stderr directives. Avoid job arrays because one Open-Science job tracks one scheduler job and one output harvest. Submit independent work as separate jobs and use the Session concurrency limit when needed. For Slurm, request resources with one `#SBATCH --option=value` directive per line (or a value-free flag such as `#SBATCH --exclusive`). The legacy `resources` option is descriptive metadata; it does not allocate CPUs, memory, or GPUs. `timeoutSeconds` limits workload runtime, not queue wait; `#SBATCH --time` sets the scheduler allocation limit. Neither is a promise of queue start time. The non-blocking job `status()` and `result()` snapshots include `scheduler_job_id` when known, `error: { code, message }` on a job-level error, and `last_poll_error` when observation or submission recovery needs attention. A pending reason or delayed accounting row does not mean the workload failed. If a submission is unconfirmed, use the reported job identity and provider diagnostics before deciding whether to submit again; Open-Science does not automatically submit a duplicate. ### Environment activation The optional `environment` value is a logical name, not a shell command. Open-Science sources `~/.open-science/environments/<name>.sh` before the workload for direct and Slurm jobs. Names are 1–64 letters, numbers, periods, underscores, or hyphens and must start with a letter or number. The file and every software/cache path it references must be visible on the execution node. If a submission reports that this activation file is missing, load the Compute Environment Setup Skill to prepare exact setup, repair, and removal instructions for the user or host administrator to run outside Open-Science. Validate the user-managed activation after they apply the plan, then retry. Do not guess a conda name, add an inline install to the science job, or hide activation in `.bashrc`. Omit `environment` when the command deliberately uses the host's default environment. ### Harvest safety boundaries - Declared output files are selected before `stdout` and `stderr`; logs use the remaining per-job budget. - The app rejects model-supplied limits above 100 MiB per file or 500 MiB per job. - Harvest also preserves a fixed 2 GiB of free local disk space. Files that do not fit remain remote. ### Behavior boundaries - **While the job runs:** the conversation is open. The user can send messages; you can handle other tasks. Each status/result query returns immediately with the current local snapshot. - **When the job finishes:** if you did not actively read its terminal result, the app initiates a new analysis turn automatically. You do not trigger this fallback. ### Check job status (non-blocking read, for informational use) ```javascript // Non-blocking DB read — no SSH. Use if you need a status snapshot mid-conversation. const handle = c.attachJob(job.job_id) const s = await handle.status() // s → { // job_id, scheduler_job_id?, status, result_final, cancellation?, exit_code, // error?, last_poll_error?, stdout_tail, stderr_tail, remote_workdir, // follow_up_delivery: 'pending' // } // status: 'queued' | 'submitted' | 'running' | 'success' | 'failed' | 'timeout' | 'cancelled' | 'error' // result_final means the automatic response is ready; partial outputs may still be remote. // error: { code, message } // cancellation: { status: 'cancelling' } or { status: 'failed', error: { code, message } } ``` To stop one active job, request durable cancellation through the same handle: ```javascript const cancellation = await c.attachJob(job.job_id).cancel() // While pending: cancellation.cancellation.status === 'cancelling'. // Completed: cancellation.status === 'cancelled', result_final === true. // Failed attempt: cancellation.cancellation.status === 'failed'; inspect its error. // An explicit cancel() on a still-active job can retry a failed cancellation. ``` Cancellation failure ends that cancellation attempt, not the computation: a snapshot can have `status: 'running'`, `result_final: false`, and `cancellation: { status: 'failed', error: ... }`. Its operation-failure notification is separate from the job's later final-result notification. `error.code`/`error.message` describe the task problem; `cancellation.error.code`/ `cancellation.error.message` describe cancellation failure. A cancelled snapshot has only top-level `status: 'cancelled'`, without a duplicate cancellation object or job error. After final `error` or `cancelled`, end waiting rather than repeatedly calling status or cancel. An `error` reports an application processing problem and does not prove the remote process stopped. Empty scheduler queries and a name-filtered `ps` check do not establish remote termination. Output files establish the bytes recovered, not the job's exit code; keep those conclusions separate. Retain `job_id`, `scheduler_job_id`, and `remote_workdir` for diagnosis. When useful output remains, inspect it before choosing a rerun based on command idempotency, compute cost, remote state, and the user's goal. See the troubleshooting reference for concrete checks. ### submitJob status values | status | meaning | | ----------- | -------------------------------------------------------------------------------------------------------- | | `queued` | waiting for a Session concurrency slot | | `submitted` | accepted; direct dispatch or Slurm queue observation is in progress | | `running` | direct process or Slurm allocation observed running | | `success` | exit code 0 | | `failed` | non-zero exit (`job_failed`) or process vanished (`process_vanished`) | | `timeout` | exceeded `timeoutSeconds` | | `cancelled` | cancellation completed; partial output collection may be incomplete | | `error` | application could not complete processing, including unavailable final status; remote work may still run | ## Workflow: the analysis turn When the app initiates the analysis turn, it provides the `job_id`, `status`, and `featured_files` (Notebook Session-relative paths under `hpc/<job_id>/featured/`). In this turn: 1. Call `attachJob(job_id).result()` to get the full result dict. 2. Inspect the outputs, run any analysis needed. 3. Call `write_artifact_file` to publish outputs worth keeping as artifacts. ```javascript // In the analysis turn — read the full harvested result (non-blocking DB + directory scan) const c = host.compute.create('ssh:<alias>') const r = await c.attachJob(job_id).result() // r → { // job_id, status, result_final, exit_code, // local_output_root: '/absolute/path/to/this/notebook/session', // producer_run_id: 'notebook-run-...', // featured_files: ['hpc/<job_id>/featured/out.result', ...], // Notebook Session-relative // hidden_files: ['hpc/<job_id>/hidden/run.log', ...],