| name | activate-workflows |
| description | Develop, test, and debug workflows on the Activate platform by Parallel Works. Use whenever the task involves an Activate / Parallel Works workflow, a remote interactive session, a web server exposed through the platform, the session_runner or script_submitter subworkflows, or the `pw` CLI (workflows create/update/run, sessions, cluster). Provides a proven local-first process plus a reference of platform facts pointing at the repo's real workflows and tutorials. |
Developing Activate (Parallel Works) workflows
A repeatable process for building a workflow that runs code on a compute resource
and (optionally) exposes a web server as a platform session. Read
references/activate-platform.md for the YAML
schema, subworkflow interfaces, pw CLI, and job-directory layout — this file is
the process; that file is the facts.
The platform docs are authoritative and updated over time — this skill is a
snapshot that can fall behind. When something here conflicts with them, trust the
docs and harden this skill (Step 5):
Assume the pw client is already authenticated — otherwise ask the user to log
in. Confirm context with pw context list if unsure. Set permissions: ['*'] in
every workflow: besides allowing everyone to run it, it is what lets the in-workflow
pw client authenticate (e.g. pw agent open-port).
Golden rules
- Reuse subworkflows for submission/sessions; plain DAGs for the rest. For a web
service + session, wrap your code with
session_runner. For "just run a
script/sim on a cluster," call script_submitter (v3.6) directly. Do not
hand-write job submission, port allocation, or tunnel logic. But pure orchestration
(multi-job DAG, data flow, fan-out) needs no subworkflow — that's a first-class
use. Learn the patterns from the repo's own workflows and tutorials, not from
invented demos (reference §9): sessions →
workflow/yamls/{webshell,jupyterlab-host,openvscode}/; job DAG / sessions / outputs
→ workflow/tutorials/nginx/; fan-out / sweep → workflow/tutorials/matrix/;
retry/failover → workflow/tutorials/round-robin-failover/.
- Pick the deployment variant by platform host — don't default to
general.
session_runner/script_submitter ship as general / emed / hsp / noaa, and
the resource/scheduler/slurm/pbs form sections differ between them. Choose:
emed for cluster.einsteinmed.edu, noaa for noaa.parallel.works, hsp for
activate.hpc.mil, general otherwise — if unclear, ask the user. Then copy the
cluster/slurm/pbs form and with: block from the matching
workflow/yamls/<service>/<variant>_v4.yaml. (pw context list shows the host.)
- Pass every required subworkflow input, and
--dry-run first. A subworkflow's
defaults-filler does NOT apply its own hidden/ignore rules, so non-optional
fields with no default must be passed explicitly even when its form would hide
them (e.g. script_submitter's cleanup_script_path). --dry-run catches this —
and variant/field mismatches — cheaply before you burn a real run.
- Develop locally before touching YAML. The YAML is a thin wrapper around code
that already works.
- Know where your job runs.
ssh.remoteHost: ${{ inputs.resource.ip }} runs a
job on that resource's login node — that's where the job dir and your process
live. See Step 4.
Step 1 — Develop the code locally first
Write and test the core program (server, simulation, script) directly on this
machine until it runs standalone. Do not write any YAML yet.
- For a web service: bind a port from an argument/env var, default to
0.0.0.0,
and serve everything the page needs. Prefer the standard library / preinstalled
tools so the controller script has nothing to install. Test every endpoint with
curl and confirm it terminates/looks right.
- Drive it the way the platform will:
python3 server.py --port 8731 … in the
background, then curl localhost:8731/.... Iterate here — it's the fastest loop.
- Keep each piece a real file (you'll check these into the repo and
checkout later).
Done when: the program runs from a clean shell with no manual setup and produces
correct output.
Step 2 — Wrap the working code in workflow YAML
Mirror the proven pattern (see workflow/yamls/webshell/general_v4.yaml): a
preprocessing job that gets your files onto the node + writes inputs.sh, then a
session_runner (or script_submitter) job.
Provide the two contract scripts session_runner expects:
controller-v3.sh — idempotent setup on the login node (has internet). Often
just verifies prerequisites.start-template-v3.sh — binds ${service_port} on 0.0.0.0, writes
${PW_PARENT_JOB_DIR}/cancel.sh (how to stop the service), and ends with
sleep inf (or runs the server in foreground).
Getting the scripts onto the node — use parallelworks/checkout, never base64
(reference §10). Two modes depending on whether Claude can push to the repo:
- Write access (recommended): ask the user to grant a deploy key with write
permission. Work on a development branch — never
main — commit and push your
code, then parallelworks/checkout that branch (branch: <dev-branch>,
sparse_checkout: [<service-dir>]). The user reviews/merges the branch, then you
flip branch: to main.
- No write access: you can't push, so stage the files on the resource (e.g.
~/pw/dev/<workflow>/) and give preprocessing two steps — the
parallelworks/checkout step commented out (configured for after the merge) and
a copy step (cp -r ~/pw/dev/<workflow>/. .) that mimics it. The user later
pushes & merges, uncomments checkout, and deletes the copy step.
Wire session_runner's service.{start_service_script,controller_script,inputs_sh, rundir} to ${PW_PARENT_JOB_DIR}/... paths, set slug ("" for a root web app),
and pass resource, cluster.scheduler, cluster.slurm, cluster.pbs from the
matching variant's *_v4.yaml. Add include-workspace: true on the
compute-clusters input if the workspace should be selectable.
Sessions served from a base-path URL. A session lives at
…/me/session/<user>/<session-name>/<slug>. Apps that build absolute URLs
(JupyterLab, many SPAs) need to know that prefix — either set the app's base-URL
(compute basepath=/me/session/${PW_USER}/${{ sessions.session }} and feed it in) or
front it with an nginx reverse proxy on ${service_port}. See reference §11 and
jupyterlab-host/start-template-v3.sh. Apps that serve relative paths just use
slug: "" and need neither.
Numbers from integer inputs arrive as strings; guard with ${var:-default}.
Step 3 — Test end-to-end with the pw client
pw workflows run --dry-run -i '{"resource":"<name>","scheduler":false}' ./my.yaml
pw workflows create my-wf --yaml my.yaml --display-name "My WF"
pw workflows update my-wf --yaml my.yaml
pw workflows run my-wf -i '{"resource":"<name>","scheduler":false,"…":…}' \
--name "e2e-1" -o json
- Pass the resource as a bare name string (
"gcpsmall", "workspace"); the
platform resolves the full object. Never hardcode IPs.
- Pick an active resource (
pw cluster ls). scheduler:false runs the service
on the login node — simplest for a demo.
- Watch progress:
pw workflows runs logs <slug> -f or poll
pw workflows runs view <slug> -o json. Success looks like the create_session
job logging "Session is ready".
- Confirm the session:
pw sessions ls -o table shows it running, tunnel, with
the remote host/port.
Step 4 — Debug from the job directory and processes
After every run, inspect artifacts on the node where the job ran (the resource's
login node when scheduler:false):
ls -la ~/pw/jobs/<workflow-name>/<NNNNN>/
cat ~/pw/jobs/<workflow-name>/<NNNNN>/run.*.out
cat ~/pw/jobs/<workflow-name>/<NNNNN>/SESSION_PORT ~/.../HOSTNAME
cat ~/pw/jobs/<workflow-name>/<NNNNN>/<your-service>.out
ps -x | grep <your-process>
curl localhost:$(cat ~/pw/jobs/<wf>/<NNNNN>/SESSION_PORT)/
The execution node may not be this machine. It's the login node of the chosen
resource. If files/processes aren't here, you targeted a different resource — either
target the resource whose login node is this host, or pw ssh <resource> to it.
Logs are always reachable via the API regardless of node:
pw workflows runs logs <slug> --job session_runner
pw workflows runs errors <slug> -o text
Diagnose → fix the local code or YAML → push to the dev branch (Mode A) or re-stage
the files (Mode B) → pw workflows update → re-run. Cancel runs you're done with:
pw workflows runs cancel <slug> (its cleanup trap runs cancel.sh, stops the
service, and removes the session). A lingering sleep inf run holds resources.
Step 5 — Harden this skill
Every time you hit a surprise — a wrong YAML field, an unexpected CLI flag, a
misunderstood subworkflow input, a debugging trick that worked — add it to this
file or the reference so the next run avoids it. These files are living documents.
Because the platform docs change, re-check them when behavior surprises you and
correct this skill toward the docs. Do not add a new tutorial to
workflow/tutorials/ without maintainer approval — each must show something new and
non-repetitive; point at an existing tutorial instead.
Best practices
- Develop locally first; prefer the standard library so controller scripts have
nothing to install.
- Deliver code with
parallelworks/checkout, not base64. Push to a dev branch and
checkout that branch (write access), or stage on the resource + a commented-out
checkout beside a stand-in copy step (no write access). See Step 2 / reference §10.
- Match the deployment variant to the platform host (
emed/noaa/hsp/general)
and copy that variant's resource/slurm/pbs form — don't reuse general blindly.
- Orchestrate with the job graph. Share files across jobs via
${PW_PARENT_JOB_DIR}
(cd into it). Pass values with echo "K=v" | tee -a $OUTPUTS (or pipe a program
that prints K=v lines) and read ${{ needs.<job>.outputs.K }}. Drive conditional
steps with if: ${{ needs.X.outputs.flag == 'true' }} — compute the boolean
upstream. Fan out with a matrix strategy (see workflow/tutorials/matrix/); fan
in with needs: [w1,w2,w3]. For retry/failover, attach a retry block to a step
and use PW_WORKFLOW_STEP_CURRENT_RETRY to pick a target per attempt — round-robin
over a list input fails over across resources (see
workflow/tutorials/round-robin-failover/; reference §3 "Step retries").
- Make scripts idempotent and traceable:
set -o pipefail, set -x; check
before installing; safe to re-run.
- Stream progress and emit structured results. Print incremental progress (it
streams to
run.<JOBID>.out / the page) and write a machine-readable result
(JSON) — don't make the user guess whether it's alive or done.
- Defensive inputs:
integer/numeric inputs arrive as strings — guard with
${var:-default}; quote every ${{ ... }} interpolation so empties/spaces don't
break the shell.
- Relative paths inside submitted scripts.
script_submitter cds into rundir
first; reference files relative to it, not via ${PW_PARENT_JOB_DIR} (which may be
unset on a SLURM/PBS compute node — the home FS is shared, so relative paths work).
- Pin subworkflow versions (
v1.4, v3.6) so a marketplace update can't silently
change behavior.
- Cross-node service reach (multi-resource workflows). When one job/service must reach
another resource's service, expose it on localhost with
pw forward -L p:host:p <resource> (host = localhost if unscheduled, the HOSTNAME value if scheduled —
see Common pitfalls). Read the other job's runtime values (its allocated port, its
HOSTNAME) with pw ssh <resource> "cat ${PW_PARENT_JOB_DIR}/<file>" — that path is the
same on every resource in a run. Reference §6.
- Gate dependent sessions without hanging. If session A needs session B's runtime
output (e.g. a dynamically-allocated port), have A wait for it, but make B fail loud
on any misconfiguration and rely on
early-cancel: any-job-failed so B's failure cancels
A — otherwise an indefinite wait hangs forever. Mirror B's values locally in A once read,
so later steps don't re-query across hosts.
- Clean up: provide
cancel.sh/cleanup scripts, and cancel runs you're done with
(pw workflows runs cancel) so nothing lingers (sleep inf, SLURM allocations).
Common pitfalls (learned from real runs)
- Wrong deployment variant / mismatched resource form: using
general's
slurm/pbs fields against an emed/noaa/hsp subworkflow fails --dry-run with
field errors. Match the variant to the host and copy its *_v4.yaml form.
- Resource passing: bare name string in
-i, not a hand-built object; login
IPs change, so never hardcode ip.
- Subworkflow "Missing required fields": pass non-optional subworkflow inputs
with no default even when its form hides them (e.g.
cleanup_script_path: "" +
define_cleanup_script: false for script_submitter).
- Scheduled jobs run on a compute node that may lack
${PW_PARENT_JOB_DIR}; use
paths relative to rundir. Cloud-burst nodes are slow to provision (6+ min
observed; idle~ → CF → RUNNING, POWERING_UP while booting) — watch with
squeue/sacct on the login node and poll with long intervals; it's not a hang.
workspace resolves with empty ip → its SSH steps run on the workspace node
(a different host than a cluster login node). Match your debugging location to
the resource you chose.- Code-delivery mistakes: don't base64-embed; if you used the no-write-access
copy step, remember it must be swapped for the (currently commented) checkout once
the branch is merged.
- Base-path apps break at the session URL if served at the host root — set the
app's base URL or front it with an nginx proxy (reference §11).
- Service must bind
0.0.0.0:${service_port} (not 127.0.0.1, not a fixed
port) or the tunnel can't reach it / the port clashes.
- Forgot
cancel.sh or sleep inf: the service is killed immediately or the
job exits before the session registers.
- Missing
permissions: ['*']: in-workflow pw calls fail to authenticate.
pw sessions stop 404s if the run was already canceled (cancel tears the
session down). Not an error.- Always
--dry-run before a real run; it catches schema/YAML problems cheaply.
pw forward target host (verified): forward a service to localhost with
pw forward -L p:host:p <resource> — host=localhost when the service is on the
login node (unscheduled; its external hostname is NOT reachable), host=<HOSTNAME file value> when it's on a compute node (scheduled). Pick by the scheduler flag.- Input group named
env + a top-level env: block referencing ${{ inputs.env.* }}
→ Expression Parser Error: max recursion exceeded, failing both --dry-run and
pw workflows run (the web UI may still submit). Rename the group (e.g. env_vars).
- Cross-cluster filesystems are separate: a code/data path staged on one resource is
absent on another — stage it on the resource that runs it. If a required path is missing,
fail loud (exit non-zero), don't silently skip: a silent skip upstream plus a
downstream job waiting on its output (e.g. a port file) becomes an indefinite hang.
Lessons from LLM-backed & multi-session builds (hermes-agent)
Reusable takeaways from building a multi-agent workflow (an orchestrator on the
workspace + a worker per cluster). Platform mechanics are in reference §12.
--dry-run is necessary but NOT sufficient. It only validates schema/variant.
"Tested end-to-end" (Step 3/4) means a real create+run, watching for "Session
is ready", exercising the live service, and debugging from ~/pw/jobs. Don't
call a workflow tested on a dry-run alone.- Test through the real client path, not a stand-in. Reach the service the way
the user will — through the platform proxy/session/chat — not just a local
curl.
Proxy-only failures (SSE streaming resets, base-path rewrites, ~60s timeouts) pass
a localhost curl and a non-streaming call, then fail in the actual UI. When a
service works locally but fails through the platform, read its own stdout/stderr
log in the job dir — the traceback there pinpoints the cause fast.
- Prove the risky/novel mechanism on real infra early, before the full workflow
is wired. For a distributed piece (e.g. cross-node
pw ssh calls), stage the
script on the target node and exercise it directly, then wrap it in YAML.
- Reuse the platform's native surfaces — don't hand-roll UI. For a chat-style
service, make it OpenAI-compatible and declare the session
openAI: true so it
joins the built-in chat. It works whether the session runs on the workspace or a
cluster — only the surface differs (verified): a workspace session shows up as
chat models in pw ai models ls; a cluster session shows up as a chat
provider in pw ai providers ls (csp: openai-tunnel), and the web Chat lists
its models under that provider. So you do NOT need a workspace proxy to chat a
cluster agent. One workspace session can also expose many models (one
/v1/models entry each, routed by the request's model) — handy for fronting
several backends from one session. SSE must be framed for the proxy. (Reference §12.)
- If a chat-style openAI service is flaky in the built-in chat, serve its own web
UI instead. Agentic/streaming cluster sessions can still report "not reachable"
(the probe hits
//HEAD/OPTIONS — answer those 200 — and slow/silent streams
flap). A web session (openAI: false, redirect: true) under the base path (often
just X-Forwarded-Prefix, no nginx) is more robust; offer both from one workflow via
a dropdown driving the sessions: block. (Verified building hermes-agent. §11/§12.)
- LLM "brain" = the platform endpoint + runtime
PW_API_KEY (+ X-Allocation
for org:* models) — no external key or org secret. To use the key in workflow
code, expose it with a top-level env: block (PW_API_KEY: ${PW_API_KEY}); keep
it out of inputs.sh (the grep -v PW_API_KEY convention) and read it from the
runtime env. (Reference §12.)
pw workflows run uses the STORED def — pw workflows update after every YAML
edit, or the run silently uses the old form.- Discover related sessions by the
sessions: key marker in the session name
(<workflow>_<n>_<key>) via pw sessions ls -o json — more robust than matching
the workflow name. (Reference §12.)
