| name | psy-exp-reviewer |
| description | Use for reviewing psychological experiment code quality at any stage — from early design idea through completed code. Supports five modes: code-audit (full platform-aware code review with smoke test protocol), config-audit (pre-code design review), implementation-plan-review (architecture review), triage-only (missing-information checklist from natural-language idea), and blocked (insufficient input). Does NOT generate or fix code. Trigger for 检查实验代码、code review、实验程序审查、能不能正式采集、代码有没有问题、check experiment code、实验设计有没有问题、帮我看看这个实验方案 / 実験コードレビュー、実験監査 / Experiment-Code-Review, Code-Audit, Experiment prufen / revue de code experimental, audit de code. |
Psychological Experiment Code Reviewer
Version
v1.3 — stable, 2026-06-10. Sub-skill of amazing-psycoder.
Purpose
Assess the quality and readiness of a psychological experiment — from early design idea through completed code. The reviewer adapts its mode to the input available. It never fabricates a readiness judgment beyond what the input supports.
This is the final mandatory gate in the experiment development chain. It evaluates code generated by psy-exp-coder, identifies issues, and enters a check → fix → re-check loop with the coder. Each audit round identifies remaining issues, the coder applies fixes, and the reviewer re-audits. This cycle repeats until zero Critical and zero Major issues remain. Only then is the final file delivered and data collection can begin. No experiment code proceeds to data collection without passing reviewer audit with ready_for_collection or ready_after_minor_fixes.
What "Production-Ready" Means
Code that passes reviewer audit must meet five conditions:
- Runs without errors — launches, displays stimuli, accepts responses, saves data, exits cleanly
- Collects correct data — all required columns present, RT measured from correct origin, accuracy coded correctly
- Does not crash mid-experiment — try/finally protects data, Escape works in every loop
- Data is analyzable — output format matches data-recording standard, NaN/timeout handled correctly
- Experiment logic is correct — trial sequence matches paradigm, response mapping unambiguous, condition ratios verified
Integration with Coder Skill
The reviewer cross-references the psy-exp-coder skill's artifacts:
| Coder artifact | Reviewer use |
|---|
| Platform spec Canonical Skeleton | Reference for correct API patterns — compare generated code against skeleton |
| Platform spec anti-pattern table | Checklist of forbidden patterns to scan for |
| Coder Quality Gate (9 items) | Minimum bar — if any gate fails, automatic not_ready_for_collection |
| Platform mapping README | Verify config→code mapping correctness |
| Paradigm reference files | Verify experiment logic (window sequence, accuracy rules) matches paradigm |
The reviewer also cross-references the psy-exp-designer skill's artifacts:
| Programming artifact | Reviewer use |
|---|
Paradigm ## Do Not Assume | Paradigm-specific checks — verify known pitfalls are addressed |
references/data-recording.md | Data output column validation — all 10 base columns present |
references/config-schema.md | Config validation rules — all 9 checks pass before code generation |
Review Modes
Before reviewing, classify the request into one mode.
| Mode | Use when | Minimum Input | Allowed Output |
|---|
code-audit | User provides experiment code | Code file or pasted code | PASS / FAIL with readiness label + platform-specific findings + smoke test protocol |
config-audit | User provides config YAML, trial timeline, or condition schema but no code | config YAML or structured experiment spec | Pre-code design review; cannot judge code correctness |
implementation-plan-review | User provides pseudocode or planned code architecture | Implementation plan | Architecture risk review; cannot judge runtime behavior |
triage-only | User provides only a natural-language experiment idea | Natural-language description | Missing-information list and design risks; cannot judge readiness |
blocked | User asks for readiness judgment but provides neither code nor config | Insufficient input | Explain what is missing; refuse to judge readiness |
If the user's input could fit multiple modes, default to the highest mode available (code-audit > config-audit > implementation-plan-review > triage-only). If input is insufficient for any productive review, use blocked.
Readiness Labels
| Label | Allowed in mode | Meaning |
|---|
ready_for_collection | code-audit only | Zero critical or major issues; code matches platform spec skeleton; smoke test passed |
ready_after_minor_fixes | code-audit only | Only minor issues remain; smoke-testable but fix before analysis |
not_ready_for_collection | code-audit, config-audit | Critical or major issues exist; do NOT collect data |
pre_code_ready | config-audit only | Config/spec complete and ready for code generation |
needs_experiment_info | triage-only, config-audit | Key design information is missing |
blocked | blocked | Cannot review; input is insufficient |
Hard rule: ready_for_collection and ready_after_minor_fixes require actual code review. If no code was provided, the highest possible label is pre_code_ready.
Scope Limitation Rule
At the start of every review output, state what was and was not reviewed. If no code was provided:
Scope: No experiment code was provided. This review cannot verify implementation details such as RT timing accuracy, keyboard handling, stimulus preloading, data saving safety, or Escape quit behavior.
Platform Detection (code-audit only)
When code is provided, first detect the platform:
| Signature | Platform |
|---|
from psychopy import / visual.Window / keyboard.Keyboard | PsychoPy |
initJsPsych / jsPsych.run / jsPsychHtmlKeyboardResponse | jsPsych |
PsychImaging / Screen('Flip' / KbQueueCreate / sca | Psychtoolbox |
Once detected, load the corresponding coder spec for authoritative API patterns:
- PsychoPy →
../psy-exp-coder/psychopy/spec/README.md
- jsPsych →
../psy-exp-coder/jspsych/spec/README.md
- Psychtoolbox →
../psy-exp-coder/psychtoolbox/spec/README.md
Also load the paradigm file from the programming layer for failure mode cross-reference:
../psy-exp-designer/paradigms/{paradigm_name}.md
If the platform cannot be detected, flag this as a critical issue.
Review Checklist — code-audit
Gate 0: Coder Quality Gate (minimum bar)
Run the 9-item Quality Gate from the coder SKILL.md first. Any failure = automatic critical issue.
| # | Check | Pass condition | How to verify |
|---|
| 1 | Skeleton match | Code structure matches platform Canonical Skeleton | Compare section-by-section: imports → params → display → preload → trial loop → data save → cleanup. Order and structure must match |
| 2 | No anti-patterns | Zero occurrences of any pattern in platform spec anti-pattern table | grep for each forbidden pattern. See §Platform Anti-Pattern Grep Patterns below |
| 3 | Spec API patterns used | API patterns in code come from platform spec canonical skeleton, not paradigms/demo files | Check RT source, keyboard API, timing loop, data save pattern against spec skeleton — not against paradigm .md examples |
| 4 | Parameters at top | All editable values in parameter block; no magic numbers | Scan for hardcoded numbers in trial loop: durations, colors, key names, file paths. Every number in trial logic must reference a variable from the params block |
| 5 | Escape in every loop | Every while loop that contains Flip/flip/frame-draw includes an escape/abort check | Count while loops in the trial section. Count escape checks. Must be equal |
| 6 | RT source | Platform-correct RT source (see §Platform-Specific Timing & RT) | Find where rt is assigned. Verify it comes from the correct source for the detected platform |
| 7 | Incremental save | Per-trial data flush; try/finally or try/catch/sca | Find the data save call. Is it inside the trial loop? Is it wrapped in try/finally? |
| 8 | Preload outside loop | No stimulus construction inside trial loop | Scan trial loop for imread, ImageStim(, MakeTexture, Sound( |
| 9 | FONT_CONFIG | CJK text → FONT_CONFIG toggle present | If text contains Chinese/Japanese/Korean characters, verify FONT_CONFIG block with OS auto-detect exists in params section |
Platform Anti-Pattern Grep Patterns
When auditing code, scan for these exact patterns per platform:
PsychoPy (14 patterns to scan):
event.getKeys( ← blocks event loop
event.waitKeys( ← blocks event loop
kb.waitKeys( ← blocks event loop in trial
time.sleep( ← blocks event loop
core.wait( ← blocks event loop (>5ms)
imread( ← inside trial loop = frame drop
ImageStim( ← inside trial loop = jitter
time.time() ← wrong RT source
clock.getTime() ← wrong RT source (use key.rt)
waitRelease=True ← adds release delay to RT
Keyboard() ← missing backend='ptb'
.exec( ← Pavlovia incompatible
preBuffer=-1 ← missing on Sound()
addLoop( ← called before loop runs (not right before)
Psychtoolbox (15 patterns to scan):
WaitSecs( ← blocks execution (except for ITI pre-trial)
KbWait ← blocking, no RT timestamp
KbCheck ← inside response loop (use KbQueueCheck)
GetSecs - stimOnset ← manual RT calculation
input( ← invisible in fullscreen
imread( ← inside trial loop
MakeTexture( ← inside trial loop
KbQueueCreate( ← inside trial loop (should be before)
KbQueueStart( ← inside trial loop (should be before)
KbQueueFlush ← MISSING at trial start
SkipSyncTests, 1 ← sync tests skipped
Sound( ← high latency audio
DrawText(..., '+' ← text-based fixation cross
Screen('Flip', w) ← missing 'when' parameter
Screen('Flip', w, 0) ← zero when parameter
jsPsych (12 patterns to scan):
jsPsych.init( ← v7 removed
'html-keyboard-response' ← string plugin type (should be class)
jsPsych.NO_KEYS ← v7 removed (use "NO_KEYS")
jsPsych.ALL_KEYS ← v7 removed (use "ALL_KEYS")
timelineVariable('x') ← missing second argument (should be true)
Date.now() ← manual RT timing
setTimeout( ← breaks event loop
setInterval( ← breaks event loop
XMLHttpRequest ← in-trial network call
fetch( ← in-trial network call
keyCode: ← hardcoded keyCode number
.select('col') ← v7 removed method
1. Experiment Logic
2. Platform-Specific Timing & RT
PsychoPy:
jsPsych:
Psychtoolbox:
3. Response Collection
4. Randomization & Conditions
5. Stimulus Validation
6. Data Integrity Verification
This section validates that the experiment produces analyzable, complete data.
6.1 Output Column Compliance
Cross-reference against data-recording.md. All 10 base columns must be present:
| # | Required Column | Check |
|---|
| 1 | subject_id | Present in output, not hardcoded |
| 2 | block | 1-indexed, increments correctly across blocks |
| 3 | trial | 1-indexed, resets or continues correctly per block |
| 4 | condition | Human-readable label (not just numeric code) |
| 5 | stimulus | Filename or stimulus ID — traceable to condition |
| 6 | correct_response | Expected key name or None for no-go |
| 7 | response | Actual key pressed or None/empty for timeout |
| 8 | rt | Float in ms, empty string '' for timeout (not 'None' or 'NaN') |
| 9 | accuracy | 1=correct, 0=incorrect, -1=timeout |
| 10 | timestamp | ISO 8601 of trial onset |
6.2 Accuracy Coding Correctness
| Trial type | Correct behavior | accuracy value |
|---|
| Go trial, correct key | response == correct_response | 1 |
| Go trial, wrong key | response != correct_response | 0 |
| Go trial, timeout | no response within deadline | -1 |
| No-go trial, withheld | no response | 1 |
| No-go trial, responded | any key pressed | 0 |
| Stop-signal, stop success | no response after stop signal | 1 |
| Stop-signal, stop fail | responded despite stop signal | 0 |
How to verify: Read the accuracy evaluation code. For no-go trials, check: if trial_type == 'no-go': accuracy = 1 if response is None else 0. A common bug is accuracy = correct_response == response which scores no-go wrong.
6.3 Crash Recovery Test
6.4 NaN/Timeout Handling
7. Emergency Quit
8. Pre-collection Readiness
9. Paradigm-Specific Failure Mode Checks
Cross-reference the paradigm file from the programming layer for known failure modes.
Load ../psy-exp-designer/paradigms/{paradigm_name}.md and check each item in ## Do Not Assume.
Go/No-go:
IAT:
Stop-signal:
N-back:
Dot-probe:
Stroop:
For paradigms without a ## Do Not Assume section: Run the generic checks:
10. Smoke Test Protocol (NEW — code-audit only)
After automated checks pass, provide the user with a concrete, step-by-step smoke test.
This is a 5-minute manual test that verifies the code actually works before real data collection.
## Smoke Test Protocol
Run this test with subject ID "test" before collecting real data.
### Test 1: Launch & Display (30 seconds)
1. Run the experiment: [exact command]
2. Verify: Window opens in fullscreen at correct resolution
3. Verify: No error messages in console
4. Press Escape → verify experiment exits cleanly (screen restored, cursor visible)
### Test 2: Full Run-through (2 minutes)
1. Run with subject ID "smoke_test_1"
2. Complete instruction screens → verify all text readable, key mappings correct
3. Complete practice trials → verify feedback displays correctly
4. Complete all formal blocks → verify block transitions work
5. Reach "thank you" screen → verify clean exit path
### Test 3: Data Output (1 minute)
1. Locate the output CSV in data/
2. Open in Excel / text editor
3. Verify: One row per trial (check row count against expected)
4. Verify: All columns present (check column headers)
5. Verify: rt column has numeric values (not empty for responded trials)
6. Verify: accuracy column has 1/0/-1 values (no unexpected codes)
### Test 4: Crash Recovery (1 minute)
1. Run with subject ID "crash_test"
2. After trial ~10, force-quit: [how to force quit on this OS]
3. Reopen the data file → verify trials 1-10 are saved with complete data
4. Verify: No corrupted/partial rows
### Test 5: Edge Cases (30 seconds)
1. Press keys NOT in the allowed set during response → verify ignored
2. Press key before stimulus onset → verify not counted as response
3. Let a trial timeout (don't press anything) → verify timeout coded correctly
4. Press Escape mid-trial → verify exits cleanly
After the user completes the smoke test, they report back. If all 5 tests pass, the code is ready_for_collection.
Review Checklist — config-audit
Only design-level checks. Do not check implementation details.
Review Checklist — implementation-plan-review
Review Checklist — triage-only
Severity Classification
| Severity | Definition | Concrete examples |
|---|
| Critical | Invalidates all data; must fix before any collection | Wrong RT measurement source (KbCheck for RT, time.time() for RT), no data save or save-only-at-end, no Escape handling, accuracy coding inverted (no-go scored as miss), KbQueue Create/Start inside trial loop, missing backend='ptb', jsPsych.init() in v7, WaitSecs() for trial timing |
| Major | Degrades data quality; fix before formal collection | Missing Escape in ITI between trials, no randomization seed, CJK font missing (tofu characters), feedback shown in formal blocks, no KbQueueFlush before trial, waitRelease=True on RT keyboard, SkipSyncTests enabled, no preBuffer=-1 on audio, stimulus loaded inside trial loop |
| Minor | Does not affect data quality; fix when convenient | Extra debug print left in code, variable naming convention, missing code comment, parameter ordering, redundant import, hardcoded path that works but should be configurable |
Output Format
Every review output follows this structure:
## Review Mode
[mode]
## Readiness Label
[label]
## Scope
[What was reviewed and what was NOT reviewed]
## Platform
[Detected platform + version assumptions]
## Overall Verdict
[PASS / PASS WITH MINOR ISSUES / FAIL]
## Critical Issues
[Must fix before collecting data — each with file path, line number, fix suggestion]
## Major Issues
[Should fix — may affect data quality — each with file path, line number, fix suggestion]
## Minor Issues
[Nice to fix — unlikely to affect results]
## Quality Gate Results
| # | Check | Result | Detail |
|---|-------|--------|--------|
| 1 | Skeleton match | PASS/FAIL | [specific mismatch if failed] |
| 2 | No anti-patterns | PASS/FAIL | [which patterns found, where] |
| ... | ... | ... | ... |
## Section-by-Section Report
[All applicable checklist sections with findings. For each failed item: what's wrong → why → how to fix]
## Paradigm-Specific Checks
[Results of paradigm failure mode cross-reference]
## Suggested Fixes
[Specific changes with file paths and line numbers. Each fix is self-contained:
"Line 87: Change `kb = keyboard.Keyboard()` to `kb = keyboard.Keyboard(backend='ptb')`"
"Line 120: Add `KbQueueFlush([], 2);` before trial loop"
"Line 156: Change `rt = GetSecs - stimOnset` to `rt = (min(firstPress(keyIdx)) - stimOnset) * 1000`"]
## Smoke Test Protocol
[Only for code-audit with PASS or PASS WITH MINOR ISSUES verdict. The standardized 5-test protocol from §10 above.]
Verdict Rules
code-audit
- PASS: Zero critical + zero major issues + smoke test passes. →
ready_for_collection
- PASS WITH MINOR ISSUES: Zero critical + zero major; minor issues only. →
ready_after_minor_fixes
- FAIL: Any critical or major issue. →
not_ready_for_collection
config-audit
- Config complete, no design issues →
pre_code_ready
- Config has gaps or design issues →
needs_experiment_info or not_ready_for_collection
implementation-plan-review
- Plan clear and complete, no platform issues →
pre_code_ready
- Plan has gaps or anti-pattern risks → recommend resolving before proceeding
blocked
No verdict given. State what input is needed to proceed and exit.
Regression Test Guide
When code has been modified after an initial review, re-audit with this focused checklist:
- Changed lines: Read the diff. What was changed?
- API pattern check: Do new/changed lines use correct platform API patterns? (Re-run Gate 0 items 2-3)
- Anti-pattern scan: Do new/changed lines introduce any anti-patterns?
- Cascade check: Did the change affect RT calculation, data saving, or escape handling?
- Smoke test re-run: Run Test 2 (full run-through) and Test 3 (data output) from the Smoke Test Protocol
If the change is trivial (e.g., parameter value only, comment fix), re-audit is optional.
If the change affects trial loop, response collection, or data saving, full re-audit is required.
First-Run Checklist (Pre-First-Subject)
Before the first real participant, the experimenter must verify:
□ Smoke test passed (all 5 tests from §10)
□ Monitor at correct resolution and refresh rate
□ Color calibration verified (if luminance-critical stimuli)
□ Audio volume tested at participant position
□ Response device tested (keyboard/button box — all response keys register)
□ Data directory writable (test with subject ID "check_write")
□ Backup plan: where is the data if the computer crashes?
□ Experimenter knows how to force-quit (Escape / Alt+F4 / power button)
□ Participant briefing script ready (explains task, key mapping, duration)
□ Debriefing script ready (explains purpose, answers questions)
The reviewer should output this checklist with every ready_for_collection verdict.
Recovery Loop
审计不是一次性报告——发现问题后进入 检查 → 修复 → 再检查 循环,直到通过。
psy-exp-coder 生成代码
│
▼
psy-exp-reviewer 审计
│
├── Critical/Major 问题 → 修复 → 重新审计
│ ↑ │
│ └─────────┘
│ 循环直到 0 Critical + 0 Major
│
└── 0 Critical + 0 Major → 通过,交付最终文件
| Issue type | Who fixes | How to fix |
|---|
| Design error (missing windows, wrong paradigm logic) | psy-exp-designer | Fix design → regenerate config → psy-exp-coder regenerate code |
| Code error (anti-pattern found, wrong API, missing escape) | psy-exp-coder | Fix code directly according to reviewer's specific findings |
| Missing data columns / wrong RT source | psy-exp-coder | Verify against platform spec → fix code |
| Missing parameter / hardcoded value | psy-exp-designer or psy-exp-coder | Fix in design or code → regenerate |
| CJK font missing / tofu characters | psy-exp-coder | Add FONT_CONFIG toggle → re-audit |
"第 N 轮审计:发现 X 个问题。修复后进入第 N+1 轮。"
每次修复后必须重新审计。审计轮次和每轮问题数记录在最终报告中。
After PASS (ready_for_collection)
When the audit passes with ready_for_collection or ready_after_minor_fixes:
"审计通过。下一步:
- 完成 First-Run Checklist 的所有项目
- 确认 Smoke Test Protocol 已通过,代码在目标机器上正常工作
- 保存审计报告作为文档
- 如需分析数据,使用
/amazing-psycoder 进入分析流水线"
Related Files