remote-compute-modal

name	remote-compute-modal
description	Run GPU jobs on the user's own Modal account via host.compute.create('byoc:modal', ...). Covers the create→submit→wait_for_notification flow, the compute_provider kernel for env setup, image/volume resolution, and the two approval cards. Load once you've decided to dispatch to Modal.
license	Apache-2.0

You're dispatching to the user's Modal account: containers spin up in Modal's cloud, on hardware you name in plain terms (gpu/cpu/memory/timeout), under their workspace, on their bill. That's the reason there are two approval cards rather than one, and the reason the env-setup surface is a separate kernel rather than something you can call inline from the control-plane kernel: each card is the user consenting to a specific, bounded use of their credential, and the architecture keeps those grants legible by keeping the surfaces apart.

If compute.create('byoc:modal', …) returns unknown provider 'byoc:modal', Modal isn't enabled in this install — ask the user to enable it under Settings → Compute → Modal (or surface the prompt; you can't enable it for them).

For first-time environment setup, see env-setup.md in this skill directory — it walks through driving the compute_provider kernel to build and record images.

Two timeouts, one timeline

Modal has two deadline timers. Both live INSIDE the sandbox, and neither can cost you the outputs of a job that ran. The timeline every job runs on:

1. Container life starts at sandbox creation — staging the inputs counts against it, so a big upload spends container time before the job runs.
2. The job runs. If it exceeds its own budget (the job timeout below), it is TERMed there.
3. Near end of container life — one harvest margin before the provider destroys the sandbox — the in-sandbox harvest watchdog TERMs whatever is still running. It runs on the sandbox's own clock, so a wrong desktop clock or a slow upload can't make it fire late.
4. After any TERM there is a grace window for checkpoint-on-TERM handlers to flush, then the process group is stopped.
5. Whatever is under ./out/ (plus the logs) is tarred and staged UNCONDITIONALLY — on success, timeout, failure, and crash alike — and harvested back into the workspace. Staging is unconditional; delivery isn't. If the stream is refused or gives up (result_rejected, harvest_failed), the staged copy waits on the sandbox, and a third, post-job timer — the idle watchdog — terminates it after ~30 min of inactivity. See "When the job fails".

The container timeout (provider_params.modal.timeout) is how long the sandbox lives; omitted, it fills from the Settings default for this provider (Settings → Compute → Modal; ceiling: Modal's 24 h platform lifetime, minus the staging margins — 85,500 s). The job timeout (timeout_seconds on submit_job) is an optional runaway guard for one job; omitted, it defaults to the container's remaining life minus the harvest margin. Name one when you know the job's budget — a hung job then costs that budget, not the whole container, and the warm container survives for the next submit.

A deadline-ended job lands as status: 'timed_out' — not a generic failure — with its partial outputs already harvested, plus a note saying the deadline (not the workload) ended the run and suggesting the remedy: a larger timeout_seconds, or resuming from a harvested checkpoint. A deadline ends the run, never the results.

Two surfaces, one provider

You reach Modal two ways. They share confinement and credentials but answer different questions, and confusing them is the most common way to waste a turn.

host.compute.create('byoc:modal', provider_params={'modal': {...}}) is the job surface. It runs in the repl tool — same as remote-compute-ssh — and is what you use for anything with inputs/outputs you want harvested into the workspace, anything on a GPU, and anything long enough that you'd want to reattach if the daemon restarts. The call itself is a stateless constructor — the tier card (provider, image, 1× A100-40GB · 8 CPU · 32 GB, container timeout, volumes) and the actual Sandbox creation both happen on the first submit_job(). submit_job/result/call_command/attach_job/close then work exactly as for SSH: .result() is non-blocking — the daemon's poller probes the sandbox, harvests out.tar.gz into hpc/<jobId>/, and emits a compute_done notification when done. To wait, exit the cell after submit_job and use the wait_for_notification brain-tool. What reaches the remote is what was in the workspace files you named, and what comes back lands under hpc/<jobId>/ through the same hardened extractor.

compute_provider({'provider': 'modal', 'code': '…'}) is the environment surface — a Python shell with the user's Modal SDK already authenticated, running in its own confined process. Use it to prepare compute: build container images JIT in the user's workspace, populate model-weight volumes by running the downloads on Modal's own infrastructure (so wide-internet fetches never traverse the local allowlist), check what assets already exist, run a short CPU smoke probe. Its first cell in a session fires the kernel card — "environment setup — build images, populate volumes, CPU probes ≤30 min; GPU jobs ask separately" — and once granted, subsequent cells in this kernel's lifetime (idle-timeout ~15 min) run without further prompts. The kernel will reject gpu= on Sandbox.create and clamp timeout to ≤30 min — the kernel's own probe budget, separate from the job surface's container timeout. It's a redirect to the job surface, not a fence; you've already been handed the credential. Terminate every sandbox you create here — sb.terminate() in the same cell, in a try/finally if you exec in between (build_env's hydrate models this). Nothing reaps it mid-session: the idle timeout kills the kernel, never your sandboxes — the ≤30-min clamp only bounds how long a forgotten one bills.

The shape of a good run is: read what's already known about this workspace, decide whether the env you need exists, build it via the compute_provider kernel if it doesn't, then run the actual job through compute.create(). The two cards appear at most once each per session — and on a warm second session with a project-level grant, neither.

Workflow

Every host.compute.* call here runs via the repl tool; the compute_provider kernel is reached via the compute_provider tool. Neither is the python tool. All three share your workspace directory but not memory — pass data through files.

Start with compute_details({provider: 'byoc:modal', mode: 'read'}). Below the orientation line is the per-workspace ledger — ### env:<name>@<spec_sha> blocks recording images that already exist in this user's Modal workspace, the volumes they pair with, and any free-text notes a previous session left. The spec_sha is the content hash of the env's source file in this binary, so a ledger entry whose hash matches is one you can use without rebuilding: pass its image ref straight to compute.create() and the kernel card never appears. A mismatch means the env definition changed in this release — the old image still works but isn't what the current env file describes, so treat it as absent.

If what you need isn't in the ledger, build it in a compute_provider cell. Two helpers are pre-bound there alongside modal and app:

• list_envs() → {name: META, …, '_envs_dir': path} for every bundled env (packages, GPU tier, secrets it needs)
• build_env(name, *, hydrate=False) → builds the image, returns {'image': 'im-…', 'spec_sha', 'volumes', 'env', 'hydrate'}

To inspect what an env actually installs (the modal.Image chain) before building, read the source from the control-plane kernel (a repl cell) — it's a skill asset:

# repl tool
print(host.skills.read('remote-compute-modal', 'envs/proteomics_jax_gpu.py')['content'])

Don't find for it from bash — the path differs across builds.

# compute_provider tool, provider='modal'
need = {'transformers', 'torch'}
envs = list_envs()
print({n: m for n, m in envs.items()
       if not n.startswith('_') and need <= set(m.get('packages', []))})
# → {'proteomics_gpu': {'packages': [...], 'gpu_default': 'A100-80GB', ...}}

r = build_env('proteomics_gpu', hydrate=True)
print(r['image'], r['spec_sha'], r['volumes'])

Carry r['image'] (the im-… reference, NOT the env name) back to the control-plane kernel (a repl cell) and pass it to provider_params.modal.image — and, for a bundled env, its NAME as provider_params.modal.env: the host then reads the env's META["egress_domains"] from the SHIPPED file at every submit, so the declaration survives rebuilds (a pointer into the shipped catalog, not a power). The two kernels share a working directory but not memory, so the simplest handoff is a workspace JSON file:

# compute_provider tool — last line of the build cell
import json; json.dump(r, open('built_env.json', 'w'))

Append a ### env:<name>@<spec_sha> block to compute_details so the next session reads the im-… ref straight from there and skips the compute_provider cell entirely. Pass r['image'] (the im-… ref) to provider_params.modal.image. The adapter will resolve a bare env name from the ledger as a fallback, but that's brittle — the ledger entry can be stale or absent. Treat the build as its own task with its own validation; don't fold it into a job submission. See env-setup.md for ad-hoc patches and gated weights.

For a quick inline call (no inputs, output is just stdout), use submit_job with no inputs. The compute_done notification payload carries status / exit_code / featured_files, plus error_kind, system_hint, and deadline_fired when set — read those disclosures from the notification itself (a deadline-truncated rc-0 job announces partial outputs there). For stdout_tail, re-enter the kernel and read it from .result() after the notification arrives (call_command is SSH-only — it bails on a byoc: handle):

# repl tool — cell ① submits and RETURNS
import json
r = json.load(open('built_env.json'))
c = host.compute.create('byoc:modal', provider_params={'modal': {
    'image': r['image'],          # the im-… ref (bare name fallback brittle)
    'env': 'proteomics_gpu',      # bundled-env NAME → its shipped egress declaration applies
    'gpu': 'A100',                # timeout omitted → container gets the Settings default
    'volumes': r['volumes'],
}})
j = c.submit_job(
    intent='load esm2_t6 on GPU',
    command='python -c "from transformers import EsmModel; import torch; '
            'm = EsmModel.from_pretrained(\\"facebook/esm2_t6_8M_UR50D\\").cuda(); '
            'print(torch.cuda.get_device_name())"',
    timeout_seconds=300,          # job guard — this probe finishes in 5 min or it's hung
)
print('JOB_ID:', j.job_id)   # ← cell ends here; kernel never blocks

Then exit the cell and use wait_for_notification. When the compute_done notification arrives, its payload has status / exit_code / featured_files, plus error_kind / system_hint / deadline_fired when set. For stdout_tail, re-enter the kernel:

# repl tool — cell ② after the compute_done notification
print(c.attach_job('<JOB_ID>').result()['stdout_tail'])  # non-blocking — poller already harvested
c.close()

For a full job with inputs and harvested outputs:

# repl tool — cell ① submits and RETURNS
c = host.compute.create('byoc:modal', provider_params={'modal': {
    'image':   r['image'],               # 'im-…' from build_env(); bare-name fallback exists but is brittle
    'env':     'proteomics_jax_gpu',     # bundled-env NAME → its shipped egress declaration applies
    'gpu':     'A100-40GB',
    'cpu':     8,
    'memory':  32768,                    # MiB — Modal's unit, not GiB
    'volumes': {'/cache': 'af2-params'}, # mount path → Modal Volume name
    'timeout': 3600,                     # container lifetime, seconds — optional; omitted → Settings default; ceiling: Modal's 24 h platform limit
}})
# → tier card → Sandbox.create() under the user's workspace → handle bound to sandbox_id

job = c.submit_job(
    intent='colabfold on target.fasta — 1× A100, ~40 min',
    command='colabfold_batch target.fasta out/ --num-recycle 3 --model-type alphafold2_ptm',
    inputs=[{'src': 'target.fasta', 'dst_filename': 'target.fasta'}],
    outputs=['out/ranked_0.pdb', {'glob': 'out/*.json', 'visibility': 'featured'},
             {'glob': '*.log', 'visibility': 'hidden'}],
    timeout_seconds=2700,                # job guard — optional; default = remaining container life − harvest margin
)
print('JOB_ID:', job.job_id)   # ← cell ends here

Then wait_for_notification — the compute_done payload includes featured_files, so you can save_artifacts(payload['featured_files']) directly without re-entering the kernel. If you want the full dict (stdout_tail / stderr_tail / job_wall_s):

# repl tool — cell ② after the compute_done notification
r = c.attach_job('<JOB_ID>').result()  # non-blocking read of compute_usage.result
save_artifacts(r['featured_files'])
c.close()

r['job_wall_s'] is the seconds run.sh ran on the remote — the actual GPU/compute duration.

timeout is the container's lifetime; timeout_seconds guards one job inside it. Both are optional: timeout fills from the Settings default, timeout_seconds from the container's remaining life minus the harvest margin. Name a timeout_seconds when you know the job's budget — a hung job then costs that budget and the warm container survives. At either deadline the job is TERMed, given grace, and its outputs are staged and harvested as on success — the job lands as status: 'timed_out' (deadline-terminated, outputs harvested), not a generic failure.

inputs= stage flat into the workdir root. dst_filename is a bare filename — 'inputs/gfp.fasta' is rejected at submit (the file would land at /work/gfp.fasta, not where the command expects). src can be a path; dst_filename can't. Omit it to default to basename(src). Need a dir layout? mkdir -p it inside command=.

Sizing inputs= — and when to use a Volume instead. Inputs are stdin-streamed through a confined path at ~0.65–0.81 MiB/s effective (measured) — budget staging time from that: 1 GiB ≈ 20–26 minutes of upload before the job starts, spent from container life. The host budgets staging from input size with a planning rate it cannot verify (upload speed isn't controllable), so a slower-than-planned upload never endangers the harvest — the watchdog's absolute-clock backstop still protects the staging window; the job just gets less runtime than nominal. If an upload dies outright, the submit fails LOUDLY with a transport error and no phantom job. Practical sizing: code, configs, and small data are what inputs= is for; it is workable up to its 1 GiB per-submit cap (slow but fine — budget the ~20–26 min of staging near the cap). Anything above 1 GiB is refused at submit time, so beyond it a Volume is the only path — and the better one well before that. Volumes (Files-tab import, or a compute_provider cell; mount via provider_params.modal.volumes) persist on Modal disks across jobs and cost zero upload per submit.

Checkpoint long jobs as they run. Write progress to ./out/ (or a Volume) periodically: at either deadline the workload gets TERM, then a grace window for checkpoint-on-TERM handlers, and whatever is under out/ is staged and harvested unconditionally. A job that checkpoints loses at most one interval to a deadline — never the run. Keep ./out/ checkpoints SMALL (≈ ≤100 MB compressed): the harvest stream back to the host runs in a bounded (~2 min) window, so a multi-GB ./out/ risks harvest_failed even though staging succeeded — large checkpoints belong on a Volume, with only small summaries under ./out/.

Only ./out/ is harvested. The wrapper tars out/ + stdout.log + stderr.log — nothing else. outputs= globs are a post-harvest filter (featured/hidden), not a what-to-collect directive. If your tool writes to cwd, a Volume mount (e.g. /atlas), or $HOME, end command= with cp -r <results> out/ (or point the tool's -o/--output at out/). A successful job with an empty out/ returns only the log paths in output_files plus a system_hint telling you so.