메뉴
3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | ssh-ray-cluster |
| description | 3-step debug loop for remote Ray cluster — submit task via SSH, check logs locally, analyze errors and fix code, repeat until resolved. |
Three-step cycle: submit -> check logs -> analyze & fix -> repeat.
Read SSH credentials and RELAX_PROJECT_ROOT from auto-memory (reference_ray_cluster_ssh.md). Ask the user if missing — never hard-code in this file.
Use paramiko to SSH into the cluster, cd to the project root, and execute the user's command.
python3 -c "
import paramiko, shlex
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST, port=PORT, username=USER, password=PASS, timeout=10)
cmd = f'cd {shlex.quote(RELAX_PROJECT_ROOT)} && <USER_COMMAND>'
try:
stdin, stdout, stderr = ssh.exec_command(cmd, timeout=60)
print(stdout.read().decode())
err = stderr.read().decode()
if err: print('STDERR:', err)
except Exception: pass # long-running commands may timeout — that's OK
finally: ssh.close()
"
Key rule: All project-relative commands (bash scripts/..., tail log/...) MUST have cd $RELAX_PROJECT_ROOT && in the same command string. Paramiko opens a fresh shell each call.
For backgrounded launches, verify separately:
pgrep -af 'ray-job.sh' | head
ray job list 2>&1 | grep RUNNING | head
The log file is on a shared filesystem mounted locally. Read it directly:
# Find the latest log
ls -lt log/<model>-*.log | head -5
# Read the tail for errors
tail -200 log/<run-name>.log
Use the Read tool on the log file path. Search for keywords: Error, Exception, Traceback, FAILED, RuntimeError, AssertionError.
Check frequency: Wait at least 1 minute between log checks. Don't poll more frequently — training jobs take minutes to hours, and frequent checks waste context.
logger.info/logger.error calls to narrow down the issue.NEVER execute these without explicit user request:
ray stop, bash scripts/tools/kill_for_ray.sh, pkill, ray job stoprm -rf on /tmp/ray/ or session directoriesOnly ray serve shutdown -y is allowed pre-submit (when user requests a relaunch).
Read-only inspection (ray status, ray job list, ray job logs, nvidia-smi, tail, grep, ps) is always safe.
Use when syncing Relax code between internal GitLab and external GitHub, especially gitlab/dev, gitlab/main, github/main, internal CR/MR handoff, linear main history, sensitive-content checks, GitHub Actions CI validation, or guarded GitHub pushes.
自动排查 Ray 调度的分布式训练任务 hang 问题。使用当训练任务无响应、资源利用率异常、任务长时间无进度时。自动收集集群状态、任务调用栈、Actor 状态，分析阻塞链条并定位根因。
Creates git commits following Conventional Commits format with type/scope/subject and detailed markdown body. Use when user wants to commit changes, create commit, save work, or stage and commit. Enforces project-specific conventions from CLAUDE.md. Each change type gets its own markdown heading (# emoji + type), with detailed item lists under each.
Diagnose Relax training launch scripts for misconfigured flags that hurt performance (time/MFU) or waste GPU memory (cards needed). Use when user asks to review/audit/check a training script, mentions "perf doctor", suspects a config is slow or OOM-prone, or wants a sanity check before launching. Produces a two-section markdown report (Performance + Memory) with cited flags, severity, and concrete fixes.
Develop and debug the Relax reinforcement learning project. Use this skill whenever modifying code in the relax/ directory, or running remote training jobs on a Ray cluster for validation. Also use it when the user mentions training, debugging training runs, submitting Ray jobs, or fixing training errors.
Expert code review of current git changes with a senior engineer lens. Detects SOLID violations, security risks, Python anti-patterns, and ML/distributed training issues. Tailored for the Relax reinforcement learning framework.