Run ado operations on remote Ray clusters using --remote execution context files. Use when the user wants to create an operation, asks about remote clusters, wants to ship local plugins or data files to a cluster, or asks about execution context YAML files. Also applies proactively when creating an operation if execution context files are present in the workspace.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Run ado operations on remote Ray clusters using --remote execution context files. Use when the user wants to create an operation, asks about remote clusters, wants to ship local plugins or data files to a cluster, or asks about execution context YAML files. Also applies proactively when creating an operation if execution context files are present in the workspace.
Running ado on remote Ray clusters
Execution context files
An execution context YAML configures a specific cluster and environment.
Multiple files can exist in the same repo for different clusters or
environments:
morrigan_execution.yaml # Morrigan cluster, standard env
vela_execution.yaml # Vela cluster
morrigan_vllm_dev_execution.yaml # Morrigan + vllm_performance from source
The file names are user-defined conventions. Check the repo root for any
*_execution.yaml files to discover what contexts are available.
When the user asks to create an operation, check the repo root for
*_execution.yaml files. If any exist, ask which (if any) they want to use
before proceeding. Do not assume remote by default โ local execution is still
common.
Do not dispatch other ado commands (get, show, create space, etc.)
remotely unless the user explicitly requests it.
Prerequisites
Before dispatching to a cluster with port-forward, verify cluster login:
oc whoami# OpenShift# or
kubectl get nodes # Kubernetes
If this fails, request user to log in first โ the port-forward will
fail with a credentials error otherwise.
Project context
The active local project context is automatically forwarded to the remote job.
To work on the same project locally and remotely, use the same active context
for both โ do not add a separate -c flag unless explicitly switching context:
# Local: uses active context
uv run ado create space -f space.yaml
# Remote: forwards the same active context automatically
uv run ado --remote morrigan_execution.yaml create operation \
-f operation.yaml --use-latest space
Only supply -c context.yaml when you need to target a different project than
the one currently active.
Operation creation command patterns
One step โ create space and operation together remotely:
Two steps โ create space locally, run operation remotely:
uv run ado create space -f space.yaml
uv run ado --remote execution_context.yaml create operation \
-f operation.yaml --use-latest space
Prefer the two-step pattern when you want the space registered in the local
metastore (e.g. for local querying or validation) before submitting.
Tips
Check if all entries under additionalFiles in the remote
execution context YAML are required for the current submission.
Comment or remove those that are not to avoid uploading
unnecessary data.
Check if the value of the wait field is suitable for
the command being executed. In general do not wait
for create operation as it can be hours long.
If you are executing get or show commands waiting is valid
as these may only take seconds to minutes.
Common Issues
file paths in YAML not valid on the remote cluster
Any file path appearing in a space, operation, or actuator configuration YAML
(e.g. mps_file, a model checkpoint, a dataset path) must satisfy both
conditions for the remote job to succeed:
File not present on cluster: add the local path to additionalFiles in
the execution context YAML.
Path invalid on cluster: use a bare filename in the YAML; ado symlinks
additionalFiles entries into the Ray working dir, so my-file.gz resolves
but /Users/me/data/my-file.gz does not.
Failing either condition produces a file-not-found error at experiment runtime,
not at submission time, so the job starts successfully but measurements fail.
Pattern to follow
To avoid, if the experiment references a file use a bare filename (no path)
in the space/operation YAML. Add the absolute local path to additionalFiles;
ado symlinks it into the Ray working directory so the bare filename resolves on
the cluster.
# space.yamlentitySpace:-identifier:mps_filepropertyDomain:variableType:OPEN_CATEGORICAL_VARIABLE_TYPEvalues:-pigeon-10.mps.gz# bare filename โ resolves from working dir
The same applies to actuator configuration files that reference local paths
(e.g. model weights, config files). Audit all -f files for local path
references before dispatching remotely.
Ray version mismatch
If you see Changing the ray version is not allowed, pin the Ray version in
fromPyPI to match the cluster:
fromPyPI:-ado-core-ray==2.52.1# match the cluster's installed version-ado-ray-tune
fromSource plugin changes not reflected in remote run
Local edits to a plugin included via fromSource may not be reflected in
the remote run. Symptoms include: a fixed import error still occurring, an
added log line not appearing, or a new parameter not being present.
The most likely cause is that the wheel built for the plugin has the same
version as a wheel already cached by Ray. Ray sees the version as already
installed and skips reinstallation.
Plugins use uv-dynamic-versioning with a format-jinja template that appends
the git node and a timestamp to every dirty or dev build (e.g.
X.Y.Z.devN+g<commit>.d<timestamp>), so each build produces a unique version.
If you are still seeing stale wheels, confirm the plugin's pyproject.toml
has the format-jinja block from the
plugin development rules.
Execution Context YAML Reference
This section is self-contained โ use it without reading any external file.
Only go to the source for topics not covered here: read
docs/user-guide/advanced/remote-execution.md if the source repo is available,
otherwise see https://ibm.github.io/ado/latest/user-guide/advanced/remote-execution/.
Minimal โ direct cluster URL
executionType:type:clusterclusterUrl:"http://ray-cluster.my-namespace.svc.cluster.local:8265"packages:fromPyPI:-ado-core-ado-ray-tune# add any other plugins requiredenvVars:PYTHONUNBUFFERED:"x"OMP_NUM_THREADS:"1"wait:false# true = stay attached until the job finishes
Port-forward cluster (OpenShift / Kubernetes)
executionType:type:clusterclusterUrl:"http://localhost:8265"# must match localPort belowportForward:namespace:my-namespaceserviceName:my-ray-cluster-head-svclocalPort:8265packages:fromPyPI:-ado-core-ado-ray-tuneenvVars:PYTHONUNBUFFERED:"x"wait:false
packages block
packages:fromPyPI:-ado-core-ado-ray-tune-ray==2.52.1# pin to match cluster version if needed-/remote/path/to/package.whl# path to a wheel already on the clusterfromSource:-plugins/actuators/my_plugin# relative to where ado --remote is run
additionalFiles โ ship local files to the cluster
additionalFiles:-/absolute/local/path/to/data_file.csv-path/to/my_data_dir/# directories also supported
Use bare filenames (no path) in space/operation YAML; ray copies
additionalFiles entries into the Ray working directory.
runtimeEnv block
runtimeEnv:setupTimeoutSeconds:1200# default 600; -1 to disableeagerInstall:false# default true
