메뉴
SWE-bench pipeline for A/B testing Grafema with Claude Code. Use when: (1) running SWE-bench experiments, (2) comparing baseline vs grafema conditions, (3) evaluating results, (4) debugging container issues.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Fix RFDB V2 graph data silently not persisting to disk after analysis. Use when: (1) rfdb-server reports "0 nodes, 0 edges" on restart despite successful analysis, (2) segment directories exist but are empty (no .bin files), (3) manifest_index.json shows total_nodes: 0 despite analysis logging 70k+ nodes, (4) Docker builds produce empty graph databases, (5) --clear flag used before analyze command. Root cause: GraphEngineV2::clear() replaces the store with MultiShardStore::ephemeral() which has path: None, causing all flush operations to write to in-memory buffers only.
Fix missing theorem proof terms when analyzing Lean 4 environments via importModules. Use when: (1) ConstantInfo.value? returns none for theorems despite TheoremVal.value being Expr, (2) building code graph / dependency extractor for Lean 4 and getting 0 proof dependency edges, (3) Lean 4.30+ project where theorem proofs appear missing from loaded environment, (4) analyzing Mathlib or any Lean 4 project and proof terms are empty. Root cause: breaking change in Lean 4.30 — value? treats theorems as opaque by default.
Fix silent metadata issues in RFDB node storage. Covers two traps: (1) metadata flattening — nested metadata fields become top-level after serialization, so node.metadata.field is undefined but node.field works. (2) reserved keys — fields named "type", "id", "name", "file", "exported" are silently stripped from metadata by _parseNode() to prevent overwriting top-level node fields.
Grafema release procedure for publishing new versions to npm. Covers happy path, pitfalls, rfdb binary lifecycle, and rollback. Use when user says "release", "publish", "bump version".
Diagnose React + Three.js (or any browser canvas) frame stutter by capturing a CDP CPU profile via Playwright instead of guessing from `[perf]` console logs. Use when: (1) frame time is bad but `[perf]` instrumentation doesn't pinpoint the offender, (2) you've spent more than ~30 minutes "fixing" suspects without measuring, (3) hot paths involve animation systems, allocation bursts, or invisible scene-graph traversal, (4) you need self-time per function with line numbers, not aggregate "render slow". The `console.warn` pattern is good for monitoring; CDP is needed for diagnosis. Covers script template, sourcemap setup so minified function names are decoded, and interactive variant where the user drives the interaction and signals stop via `touch /tmp/<flag>`.
Force `rust-embed` to re-embed assets from `$GRAFEMA_UI_DIST` (or any env-driven path) when cargo's incremental compilation skips the macro re-expansion. Use when: (1) you rebuilt the GUI bundle but the rust-server binary still serves the OLD bundle, (2) `cargo build --release` finishes in seconds (incremental hit) and `strings target/release/binary | grep <new-hash>` finds nothing, (3) any rust-embed pipeline where assets live outside the cargo source tree and are picked up via env var or symlink. Cargo only re-runs the proc macro when the rust file declaring `#[derive(RustEmbed)]` changes — env var changes alone don't trigger it.
| name | swe-bench-grafema-experiments |
| description | SWE-bench pipeline for A/B testing Grafema with Claude Code. Use when: (1) running SWE-bench experiments, (2) comparing baseline vs grafema conditions, (3) evaluating results, (4) debugging container issues. |
| author | Claude Code |
| version | 3.0.0 |
| date | "2026-03-17T00:00:00.000Z" |
Uses claude -p inside SWE-bench Docker containers. Two conditions:
grafema installed with pre-built graph + MCP toolsSame agent, same environment, same prompt (except tool docs section).
# Generate tasks.json (one-time)
python scripts/swe-bench/generate-tasks.py > scripts/swe-bench/tasks.json
# Run single task
./scripts/swe-bench/run.sh axios__axios-4731 --mode baseline
./scripts/swe-bench/run.sh axios__axios-4731 --mode grafema
# Run all JS/TS tasks (loop)
for task in $(jq -r '.[].instance_id' scripts/swe-bench/tasks.json); do
./scripts/swe-bench/run.sh "$task" --mode baseline
./scripts/swe-bench/run.sh "$task" --mode grafema
done
# Compare results
./scripts/swe-bench/compare.sh
# Evaluate with swebench
source /Users/vadimr/swe-bench-research/mini-swe-agent/.venv/bin/activate
python -m swebench.harness.run_evaluation \
--dataset_name swe-bench/SWE-Bench_Multilingual \
--predictions_path scripts/swe-bench/results/baseline/preds.jsonl \
--max_workers 1 --run_id baseline
| File | Purpose |
|---|---|
scripts/swe-bench/run.sh | Main pipeline script |
scripts/swe-bench/compare.sh | Results comparison |
scripts/swe-bench/generate-tasks.py | Task generation from HuggingFace |
scripts/swe-bench/tasks.json | Pre-cached 43 JS/TS tasks |
scripts/swe-bench/templates/prompt-baseline.md | Baseline prompt |
scripts/swe-bench/templates/prompt-grafema.md | Grafema prompt (with tool docs) |
_ai/swe-bench-runbook.md | Full runbook |
# Check image exists
docker images | grep sweb.eval
# Check Node version
docker run --rm <image> node --version
# Debug inside container
docker run -it --name debug-swe -v ~/.claude:/root/.claude:ro <image> bash
# Verify host auth
claude --version
# Check mount inside container
docker exec <container> ls -la /root/.claude/
# Check if grafema installed
docker exec <container> grafema --version
# Check if graph was built
docker exec <container> ls /testbed/.grafema/
# Check MCP config
docker exec <container> cat /testbed/.mcp.json
Grafema consistently reduces file exploration (39-100%) but hasn't improved fix correctness in tested tasks. The new claude -p pipeline should provide cleaner measurements.