Run any Skill in Manus with one click

ssh-ray-cluster

Stars470

Forks59

UpdatedJune 3, 2026 at 13:15

3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

redai-infra

redai-infra/Relax

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	ssh-ray-cluster
description	3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved.

SSH Debug Loop

Three-step cycle: submit -> check logs -> analyze & fix -> repeat.

Prerequisites

Read SSH credentials and RELAX_PROJECT_ROOT from auto-memory (reference_ray_cluster_ssh.md). Ask the user if missing — never hard-code in this file.

Step 1: Submit Task via SSH

Use paramiko to SSH into the cluster, cd to the project root, and execute the user's command.

python3 -c "
import paramiko, shlex
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, port=PORT, username=USER, password=PASS, timeout=10)
cmd = f'cd {shlex.quote(RELAX_PROJECT_ROOT)} && <USER_COMMAND>'
try:
    stdin, stdout, stderr = ssh.exec_command(cmd, timeout=60)
    print(stdout.read().decode())
    err = stderr.read().decode()
    if err: print('STDERR:', err)
except Exception: pass  # long-running commands may timeout — that's OK
finally: ssh.close()
"

Key rule: All project-relative commands (bash scripts/..., tail log/...) MUST have cd $RELAX_PROJECT_ROOT && in the same command string. Paramiko opens a fresh shell each call.

For backgrounded launches, verify separately:

pgrep -af 'ray-job.sh' | head
ray job list 2>&1 | grep RUNNING | head

Step 2: Check Logs Locally

The log file is on a shared filesystem mounted locally. Read it directly:

# Find the latest log
ls -lt log/<model>-*.log | head -5

# Read the tail for errors
tail -200 log/<run-name>.log

Use the Read tool on the log file path. Search for keywords: Error, Exception, Traceback, FAILED, RuntimeError, AssertionError.

Check frequency: Wait at least 1 minute between log checks. Don't poll more frequently — training jobs take minutes to hours, and frequent checks waste context.

Step 3: Analyze & Fix

Identify the error from the log (traceback, error message, hang pattern).
Fix the code if the root cause is clear — edit the source file directly.
Add debug logging if the root cause is unclear — add targeted logger.info/logger.error calls to narrow down the issue.
Go back to Step 1 — resubmit the task and repeat until resolved.

Safety Rules

NEVER execute these without explicit user request:

ray stop, bash scripts/tools/kill_for_ray.sh, pkill, ray job stop
rm -rf on /tmp/ray/ or session directories

Only ray serve shutdown -y is allowed pre-submit (when user requests a relaunch).

Read-only inspection (ray status, ray job list, ray job logs, nvidia-smi, tail, grep, ps) is always safe.

More from this repository

same repository

sync-github

redai-infra/Relax

Use when syncing Relax code between internal GitLab and external GitHub, especially gitlab/dev, gitlab/main, github/main, internal CR/MR handoff, linear main history, sensitive-content checks, GitHub Actions CI validation, or guarded GitHub pushes.

2026-07-11470

debug-hang

redai-infra/Relax

自动排查 Ray 调度的分布式训练任务 hang 问题。使用当训练任务无响应、资源利用率异常、任务长时间无进度时。自动收集集群状态、任务调用栈、Actor 状态，分析阻塞链条并定位根因。

2026-06-13470

git-commit

redai-infra/Relax

Creates git commits following Conventional Commits format with type/scope/subject and detailed markdown body. Use when user wants to commit changes, create commit, save work, or stage and commit. Enforces project-specific conventions from CLAUDE.md. Each change type gets its own markdown heading (# emoji + type), with detailed item lists under each.

2026-06-03470

perf-doctor

redai-infra/Relax

Diagnose Relax training launch scripts for misconfigured flags that hurt performance (time/MFU) or waste GPU memory (cards needed). Use when user asks to review/audit/check a training script, mentions "perf doctor", suspects a config is slow or OOM-prone, or wants a sanity check before launching. Produces a two-section markdown report (Performance + Memory) with cited flags, severity, and concrete fixes.

2026-05-29470

relax-dev-debug

redai-infra/Relax

Develop and debug the Relax reinforcement learning project. Use this skill whenever modifying code in the relax/ directory, or running remote training jobs on a Ray cluster for validation. Also use it when the user mentions training, debugging training runs, submitting Ray jobs, or fixing training errors.

2026-05-14470

code-review

redai-infra/Relax

Expert code review of current git changes with a senior engineer lens. Detects SOLID violations, security risks, Python anti-patterns, and ML/distributed training issues. Tailored for the Relax reinforcement learning framework.

2026-04-14470

name	ssh-ray-cluster
description	3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved.

SSH Debug Loop

Three-step cycle: submit -> check logs -> analyze & fix -> repeat.

Prerequisites

Read SSH credentials and RELAX_PROJECT_ROOT from auto-memory (reference_ray_cluster_ssh.md). Ask the user if missing — never hard-code in this file.

Step 1: Submit Task via SSH

Use paramiko to SSH into the cluster, cd to the project root, and execute the user's command.

python3 -c "
import paramiko, shlex
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, port=PORT, username=USER, password=PASS, timeout=10)
cmd = f'cd {shlex.quote(RELAX_PROJECT_ROOT)} && <USER_COMMAND>'
try:
    stdin, stdout, stderr = ssh.exec_command(cmd, timeout=60)
    print(stdout.read().decode())
    err = stderr.read().decode()
    if err: print('STDERR:', err)
except Exception: pass  # long-running commands may timeout — that's OK
finally: ssh.close()
"

Key rule: All project-relative commands (bash scripts/..., tail log/...) MUST have cd $RELAX_PROJECT_ROOT && in the same command string. Paramiko opens a fresh shell each call.

For backgrounded launches, verify separately:

pgrep -af 'ray-job.sh' | head
ray job list 2>&1 | grep RUNNING | head

Step 2: Check Logs Locally

The log file is on a shared filesystem mounted locally. Read it directly:

# Find the latest log
ls -lt log/<model>-*.log | head -5

# Read the tail for errors
tail -200 log/<run-name>.log

Use the Read tool on the log file path. Search for keywords: Error, Exception, Traceback, FAILED, RuntimeError, AssertionError.

Check frequency: Wait at least 1 minute between log checks. Don't poll more frequently — training jobs take minutes to hours, and frequent checks waste context.

Step 3: Analyze & Fix

Identify the error from the log (traceback, error message, hang pattern).
Fix the code if the root cause is clear — edit the source file directly.
Add debug logging if the root cause is unclear — add targeted logger.info/logger.error calls to narrow down the issue.
Go back to Step 1 — resubmit the task and repeat until resolved.

Safety Rules

NEVER execute these without explicit user request:

ray stop, bash scripts/tools/kill_for_ray.sh, pkill, ray job stop
rm -rf on /tmp/ray/ or session directories

Only ray serve shutdown -y is allowed pre-submit (when user requests a relaunch).

Read-only inspection (ray status, ray job list, ray job logs, nvidia-smi, tail, grep, ps) is always safe.