remote-compute-modal

You're dispatching to the user's Modal account: containers spin up in Modal's cloud, on hardware you name in plain terms (gpu/cpu/memory/timeout), under their workspace, on their bill. That's the reason there are two approval cards rather than one, and the reason the env-setup surface is a separate kernel rather than something you can call inline from the control-plane kernel: each card is the user consenting to a specific, bounded use of their credential, and the architecture keeps those grants legible by keeping the surfaces apart. If `compute.create('byoc:modal', …)` returns `unknown provider 'byoc:modal'`, Modal isn't enabled in this install — ask the user to enable it under *Settings → Compute → Modal* (or surface the prompt; you can't enable it for them). For first-time environment setup, see `env-setup.md` in this skill directory — it walks through driving the `compute_provider` kernel to build and record images. ## Two timeouts, one timeline Modal has two deadline timers. Both live INSIDE the sandbox, and neither can cost you the outputs of a job that ran. The timeline every job runs on: 1. Container life starts at sandbox creation — staging the inputs counts against it, so a big upload spends container time before the job runs. 2. The job runs. If it exceeds its own budget (the job timeout below), it is TERMed there. 3. Near end of container life — one harvest margin before the provider destroys the sandbox — the in-sandbox **harvest watchdog** TERMs whatever is still running. It runs on the sandbox's own clock, so a wrong desktop clock or a slow upload can't make it fire late. 4. After any TERM there is a grace window for checkpoint-on-TERM handlers to flush, then the process group is stopped. 5. Whatever is under `./out/` (plus the logs) is tarred and staged UNCONDITIONALLY — on success, timeout, failure, and crash alike — and harvested back into the workspace. Staging is unconditional; delivery isn't. If the stream is refused or gives up (`result_rejected`, `harvest_failed`), the staged copy waits on the sandbox, and a third, post-job timer — the **idle watchdog** — terminates it after ~30 min of inactivity. See "When the job fails". The **container timeout** (`provider_params.modal.timeout`) is how long the sandbox lives; omitted, it fills from the Settings default for this provider (*Settings → Compute → Modal*; ceiling: Modal's 24 h platform lifetime, minus the staging margins — 85,500 s). The **job timeout** (`timeout_seconds` on `submit_job`) is an optional runaway guard for one job; omitted, it defaults to the container's remaining life minus the harvest margin. Name one when you know the job's budget — a hung job then costs that budget, not the whole container, and the warm container survives for the next submit. A deadline-ended job lands as `status: 'timed_out'` — not a generic failure — with its partial outputs already harvested, plus a note saying the deadline (not the workload) ended the run and suggesting the remedy: a larger `timeout_seconds`, or resuming from a harvested checkpoint. A deadline ends the run, never the results. ## Two surfaces, one provider You reach Modal two ways. They share confinement and credentials but answer different questions, and confusing them is the most common way to waste a turn. `host.compute.create('byoc:modal', provider_params={'modal': {...}})` is the **job surface**. It runs in the `repl` tool — same as `remote-compute-ssh` — and is what you use for anything with inputs/outputs you want harvested into the workspace, anything on a GPU, and anything long enough that you'd want to reattach if the daemon restarts. The call itself is a stateless constructor — the **tier card** (provider, image, `1× A100-40GB · 8 CPU · 32 GB`, container timeout, volumes) and the actual Sandbox creation both happen on the first `submit_job()`. `submit_job`/`result`/`call_command`/`attach_job`/`close` then work exactly as for SSH: `.result()` is **non-blocking** — the daemon's poller probes the sandbox, harvests `out.tar.gz` into `hpc/<jobId>/`, and emits a `compute_done` notification when done. To wait, exit the cell after `submit_job` and use the `wait_for_notification` brain-tool. What reaches the remote is what was in the workspace files you named, and what comes back lands under `hpc/<jobId>/` through the same hardened extractor. `compute_provider({'provider': 'modal', 'code': '…'})` is the **environment surface** — a Python shell with the user's Modal SDK already authenticated, running in its own confined process. Use it to *prepare* compute: build container images JIT in the user's workspace, populate model-weight volumes by running the downloads on Modal's own infrastructure (so wide-internet fetches never traverse the local allowlist), check what assets already exist, run a short CPU smoke probe. Its first cell in a session fires the **kernel card** — *"environment setup — build images, populate volumes, CPU probes ≤30 min; GPU jobs ask separately"* — and once granted, subsequent cells in this kernel's lifetime (idle-timeout ~15 min) run without further prompts. The kernel will reject `gpu=` on `Sandbox.create` and clamp `timeout` to ≤30 min — the kernel's own probe budget, separate from the job surface's container timeout. It's a redirect to the job surface, not a fence; you've already been handed the credential. **Terminate every sandbox you create here** — `sb.terminate()` in the same cell, in a `try/finally` if you `exec` in between (`build_env`'s hydrate models this). Nothing reaps it mid-session: the idle timeout kills the *kernel*, never your sandboxes — the ≤30-min clamp only bounds how long a forgotten one bills. The shape of a good run is: read what's already known about this workspace, decide whether the env you need exists, build it via the `compute_provider` kernel if it doesn't, then run the actual job through `compute.create()`. The two cards appear at most once each per session — and on a warm second session with a project-level grant, neither. ## Workflow Every `host.compute.*` call here runs via the **`repl` tool**; the `compute_provider` kernel is reached via the **`compute_provider` tool**. Neither is the `python` tool. All three share your workspace directory but not memory — pass data through files. Start with `compute_details({provider: 'byoc:modal', mode: 'read'})`. Below the orientation line is the per-workspace ledger — `### env:<name>@<spec_sha>` blocks recording images that already exist in this user's Modal workspace, the volumes they pair with, and any free-text notes a previous session left. The `spec_sha` is the content hash of the env's source file in this binary, so a ledger entry whose hash matches is one you can use without rebuilding: pass its image ref straight to `compute.create()` and the kernel card never appears. A mismatch means the env definition changed in this release — the old image still works but isn't what the current env file describes, so treat it as absent. If what you need isn't in the ledger, build it in a `compute_provider` cell. Two helpers are pre-bound there alongside `modal` and `app`: - `list_envs()` → `{name: META, …, '_envs_dir': path}` for every bundled env (packages, GPU tier, secrets it needs) - `build_env(name, *, hydrate=False)` → builds the image, returns `{'image': 'im-…', 'spec_sha', 'volumes', 'env', 'hydrate'}` To inspect what an env actually installs (the `modal.Image` chain) before building, read the source from the **control-plane kernel** (a `repl` cell) — it's a skill asset: ```python # repl tool print(host.skills.read('remote-compute-modal', 'envs/proteomics_jax_gpu.py')['content']) ``` Don't `find` for it from bash — the path differs across builds. ```python # compute_provider tool, provider='modal' need = {'transformers', 'torch'} envs = list_envs() print({n: m for n, m in envs.items() if not n.startswith('_') and need <= set(m.get('packages', []))}) # → {'proteomics_gpu': {'packages': [...], 'gpu_default': 'A100-80GB', ...}} r = build_env('proteomics_gpu', hydrate=True) print(r['image'], r['spec_sha'], r['volumes']) ``` Carry `r['image']` (the `im-…` reference, NOT the env name) back to the control-plane kernel (a `repl` cell) and pass it to `provider_params.modal.image` — and, for a bundled env, its NAME as `provider_params.modal.env`: the host then reads the env's `META["egress_domains"]` from the SHIPPED file at every submit, so the declaration survives rebuilds (a pointer into the shipped catalog, not a power). The two kernels share a working directory but not memory, so the simplest handoff is a workspace JSON file: ```python # compute_provider tool — last line of the build cell import json; json.dump(r, open('built_env.json', 'w')) ``` Append a `### env:<name>@<spec_sha>` block to `compute_details` so the next session reads the `im-…` ref straight from there and skips the `compute_provider` cell entirely. **Pass `r['image']` (the `im-…` ref) to `provider_params.modal.image`.** The adapter will resolve a bare env name from the ledger as a fallback, but that's brittle — the ledger entry can be stale or absent. Treat the build as its own task with its own validation; don't fold it into a job submission. See `env-setup.md` for ad-hoc patches and gated weights. For a quick inline call (no inputs, output is just stdout), use `submit_job` with no `inputs`. The `compute_done` notification payload carries `status` / `exit_code` / `featured_files`, plus `error_kind`, `system_hint`, and `deadline_fired` when set — read those disclosures from the notification itself (a deadline-truncated rc-0 job announces partial outputs there). For `stdout_tail`, re-enter the kernel and read it from `.result()` after the notification arrives (`call_command` is SSH-only — it bails on a `byoc:` handle): ```python # repl tool — cell ① submits and RETURNS import json r = json.load(open('built_env.json')) c = host.compute.create('byoc:modal', provider_params={'modal': { 'image': r['image'], # the im-… ref (bare name fallback brittle) 'env': 'proteomics_gpu', # bundled-env NAME → its shipped egress declaration applies 'gpu': 'A100', # timeout omitted → container gets the Settings default 'volumes': r['volumes'], }}) j = c.submit_job( intent='load esm2_t6 on GPU', command='python -c "from transformers import EsmModel; import torch; ' 'm = EsmModel.from_pretrained(\\"facebook/esm2_t6_8M_UR50D\\").cuda(); ' 'print(torch.cuda.get_device_name())"', timeout_seconds=300, # job guard — this probe finishes in 5 min or it's hung ) print('JOB_ID:', j.job_id) # ← cell ends here; kernel never blocks ``` Then exit the cell and use `wait_for_notification`. When the `compute_done` notification arrives, its payload has `status` / `exit_code` / `featured_files`, plus `error_kind` / `system_hint` / `deadline_fired` when set. For `stdout_tail`, re-enter the kernel: ```python # repl tool — cell ② after the compute_done notification print(c.attach_job('<JOB_ID>').result()['stdout_tail']) # non-blocking — poller already harvested c.close() ``` For a full job with inputs and harvested outputs: ```python # repl tool — cell ① submits and RETURNS c = host.compute.create('byoc:modal', provider_params={'modal': { 'image': r['image'], # 'im-…' from build_env(); bare-name fallback exists but is brittle 'env': 'proteomics_jax_gpu', # bundled-env NAME → its shipped egress declaration applies 'gpu': 'A100-40GB', 'cpu': 8, 'memory': 32768, # MiB — Modal's unit, not GiB 'volumes': {'/cache': 'af2-params'}, # mount path → Modal Volume name 'timeout': 3600, # container lifetime, seconds — optional; omitted → Settings default; ceiling: Modal's 24 h platform limit }}) # → tier card → Sandbox.create() under the user's workspace → handle bound to sandbox_id job = c.submit_job( intent='colabfold on target.fasta — 1× A100, ~40 min', command='colabfold_batch target.fasta out/ --num-recycle 3 --model-type alphafold2_ptm', inputs=[{'src': 'target.fasta', 'dst_filename': 'target.fasta'}], outputs=['out/ranked_0.pdb', {'glob': 'out/*.json', 'visibility': 'featured'}, {'glob': '*.log', 'visibility': 'hidden'}], timeout_seconds=2700, # job guard — optional; default = remaining container life − harvest margin ) print('JOB_ID:', job.job_id) # ← cell ends here ``` Then `wait_for_notification` — the `compute_done` payload includes `featured_files`, so you can `save_artifacts(payload['featured_files'])` directly without re-entering the kernel. If you want the full dict (`stdout_tail` / `stderr_tail` / `job_wall_s`): ```python # repl tool — cell ② after the compute_done notification r = c.attach_job('<JOB_ID>').result() # non-blocking read of compute_usage.result save_artifacts(r['featured_files']) c.close() ``` `r['job_wall_s']` is the seconds `run.sh` ran on the remote — the actual GPU/compute duration. `timeout` is the container's lifetime; `timeout_seconds` guards one job inside it. Both are optional: `timeout` fills from the Settings default, `timeout_seconds` from the container's remaining life minus the harvest margin. Name a `timeout_seconds` when you know the job's budget — a hung job then costs that budget and the warm container survives. At either deadline the job is TERMed, given grace, and its outputs are staged and harvested as on success — the job lands as `status: 'timed_out'` (deadline-terminated, outputs harvested), not a generic failure. `inputs=` stage **flat** into the workdir root. `dst_filename` is a bare filename — `'inputs/gfp.fasta'` is rejected at submit (the file would land at `/work/gfp.fasta`, not where the command expects). `src` can be a path; `dst_filename` can't. Omit it to default to `basename(src)`. Need a dir layout? `mkdir -p` it inside `command=`. **Sizing `inputs=` — and when to use a Volume instead.** Inputs are stdin-streamed through a confined path at ~0.65–0.81 MiB/s effective (measured) — budget staging time from that: 1 GiB ≈ 20–26 minutes of upload before the job starts, spent from container life. The host budgets staging from input size with a planning rate it cannot verify (upload speed isn't controllable), so a slower-than-planned upload never endangers the harvest — the watchdog's absolute-clock backstop still protects the staging window; the job just gets less runtime than nominal. If an upload dies outright, the submit fails LOUDLY with a transport error and no phantom job. Practical sizing: code, configs, and small data are what `inputs=` is for; it is workable up to its **1 GiB per-submit cap** (slow but fine — budget the ~20–26 min of staging near the cap). Anything above 1 GiB is refused at submit time, so beyond it a Volume is the only path — and the better one well before that. Volumes (Files-tab import, or a `compute_provider` cell; mount via `provider_params.modal.volumes`) persist on Modal disks across jobs and cost zero upload per submit. **Checkpoint long jobs as they run.** Write progress to `./out/` (or a Volume) periodically: at either deadline the workload gets TERM, then a grace window for checkpoint-on-TERM handlers, and whatever is under `out/` is staged and harvested unconditionally. A job that checkpoints loses at most one interval to a deadline — never the run. Keep `./out/` checkpoints SMALL (≈ ≤100 MB compressed): the harvest stream back to the host runs in a bounded (~2 min) window, so a multi-GB `./out/` risks `harvest_failed` even though staging succeeded — large checkpoints belong on a Volume, with only small summaries under `./out/`. **Only `./out/` is harvested.** The wrapper tars `out/` + `stdout.log` + `stderr.log` — nothing else. `outputs=` globs are a *post-harvest* filter (featured/hidden), not a what-to-collect directive. If your tool writes to cwd, a Volume mount (e.g. `/atlas`), or `$HOME`, end `command=` with `cp -r <results> out/` (or point the tool's `-o`/`--output` at `out/`). A successful job with an empty `out/` returns only the log paths in `output_files` plus a `system_hint` telling you so.