القائمة
3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | ssh-ray-cluster |
| description | 3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved. |
Three-step cycle: submit -> check logs -> analyze & fix -> repeat.
Read SSH credentials and RELAX_PROJECT_ROOT from auto-memory (reference_ray_cluster_ssh.md). Ask the user if missing — never hard-code in this file.
Use paramiko to SSH into the cluster, cd to the project root, and execute the user's command.
python3 -c "
import paramiko, shlex
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, port=PORT, username=USER, password=PASS, timeout=10)
cmd = f'cd {shlex.quote(RELAX_PROJECT_ROOT)} && <USER_COMMAND>'
try:
stdin, stdout, stderr = ssh.exec_command(cmd, timeout=60)
print(stdout.read().decode())
err = stderr.read().decode()
if err: print('STDERR:', err)
except Exception: pass # long-running commands may timeout — that's OK
finally: ssh.close()
"
Key rule: All project-relative commands (bash scripts/..., tail log/...) MUST have cd $RELAX_PROJECT_ROOT && in the same command string. Paramiko opens a fresh shell each call.
For backgrounded launches, verify separately:
pgrep -af 'ray-job.sh' | head
ray job list 2>&1 | grep RUNNING | head
The log file is on a shared filesystem mounted locally. Read it directly:
# Find the latest log
ls -lt log/<model>-*.log | head -5
# Read the tail for errors
tail -200 log/<run-name>.log
Use the Read tool on the log file path. Search for keywords: Error, Exception, Traceback, FAILED, RuntimeError, AssertionError.
Check frequency: Wait at least 1 minute between log checks. Don't poll more frequently — training jobs take minutes to hours, and frequent checks waste context.
logger.info/logger.error calls to narrow down the issue.NEVER execute these without explicit user request:
ray stop, bash scripts/tools/kill_for_ray.sh, pkill, ray job stoprm -rf on /tmp/ray/ or session directoriesOnly ray serve shutdown -y is allowed pre-submit (when user requests a relaunch).
Read-only inspection (ray status, ray job list, ray job logs, nvidia-smi, tail, grep, ps) is always safe.
Use when syncing Relax code between internal GitLab and external GitHub, especially gitlab/dev, gitlab/main, github/main, internal CR/MR handoff, linear main history, sensitive-content checks, GitHub Actions CI validation, or guarded GitHub pushes.
自动排查 Ray 调度的分布式训练任务 hang 问题。使用当训练任务无响应、资源利用率异常、任务长时间无进度时。自动收集集群状态、任务调用栈、Actor 状态，分析阻塞链条并定位根因。
Creates git commits following Conventional Commits format with type/scope/subject and detailed markdown body. Use when user wants to commit changes, create commit, save work, or stage and commit. Enforces project-specific conventions from CLAUDE.md. Each change type gets its own markdown heading (# emoji + type), with detailed item lists under each.
Diagnose Relax training launch scripts for misconfigured flags that hurt performance (time/MFU) or waste GPU memory (cards needed). Use when user asks to review/audit/check a training script, mentions "perf doctor", suspects a config is slow or OOM-prone, or wants a sanity check before launching. Produces a two-section markdown report (Performance + Memory) with cited flags, severity, and concrete fixes.
Develop and debug the Relax reinforcement learning project. Use this skill whenever modifying code in the relax/ directory, or running remote training jobs on a Ray cluster for validation. Also use it when the user mentions training, debugging training runs, submitting Ray jobs, or fixing training errors.
Expert code review of current git changes with a senior engineer lens. Detects SOLID violations, security risks, Python anti-patterns, and ML/distributed training issues. Tailored for the Relax reinforcement learning framework.