add-unit-test

name

add-unit-test

description

Add LLVM-IR unit tests for a transform pass in this team compiler under `test/unit/<PassName>/`. Triggers on `/add-unit-test` or any request to "add unit tests / 유닛 테스트 추가 / 테스트 작성" for a pass in this repo. Enforces the team's mandatory rule: **at least 3 tests, and at least one must be a real-IR snippet from `swpp202601-benchmarks/<bench>/src/<bench>.ll` where the pass actually fires**, with a comment showing the approximate TM cost delta (before/after) so both correctness and effectiveness are visible at the unit-test layer. Also drafts the matching cost analysis line for the PR description.

add-unit-test — Author benchmark-grounded unit tests for a transform pass

When this skill is invoked, produce a complete, runnable unit-test set for the requested pass. The output must satisfy the team's grading rule (Guide/Spec/UnitTestGuide.md):

최소 3개의 unit test. 그 중 하나는 swpp202601-benchmarks/{bench}/src/{bench}.ll 에서 발췌한 실제 IR을 사용. TM spec 기준 cost 감소를 PR에 명시.

This skill encodes that rule and the surrounding craft.

Step 0 — Identify the pass

Ask the user (or infer from the diff / current branch) which pass needs tests. Confirm:

• Pass class name (e.g. ShiftToMul, InstCombine).
• Test directory: test/unit/<PassName>/.
• Read the pass's source under lib/ (or wherever it lives) so you know what patterns it actually rewrites and what it leaves alone. Without this, you cannot tell which benchmark snippets will fire.

If the pass already has tests, list the existing ones first and design new tests to fill gaps — do not duplicate coverage.

Step 1 — Find a benchmark IR snippet where the pass fires

Benchmark .ll files are pre-baked inside the CI image (ghcr.io/dennis0405/swpp202601-team03-ci:latest) at /opt/benchmarks/*/src/*.ll. Use the existing container if one is stopped or running (same approach as bench-cost):

# Check for an existing CI container
docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" | grep swpp202601-team03-ci

# Start it and grep inside
docker start <container_name>
docker exec <container_name> grep -rn 'shl i.* %' /opt/benchmarks/*/src/*.ll

# Or read a specific file
docker exec <container_name> cat /opt/benchmarks/matmul1/src/matmul1.ll

If no container exists yet, run without one:

docker run --rm \
  ghcr.io/dennis0405/swpp202601-team03-ci:latest \
  grep -rn 'shl i.* %' /opt/benchmarks/*/src/*.ll

Pick a candidate by these criteria, in order:

1. Pass actually fires on the snippet (verify by reading the pass logic — do not guess).
2. Snippet is small enough to lift into a self-contained .ll test (typically a single function or a single loop body).
3. Effect is measurable — the rewritten IR should have a non-trivial TM-cost delta. A shl-vs-mul swap inside a hot loop body wins big; a one-shot snippet outside any loop wins little.
4. Prefer benchmarks whose name appears in the project's competition set (matmul*, rmq*, bubble_sort, etc.) over edge-case benches.

Cite the source explicitly in the test file header:

; Source: swpp202601-benchmarks/<bench>/src/<bench>.ll  (lines N..M, function @foo)

If you copy a whole function, keep the original name. If you carve out a loop body, wrap it in a synthetic function — the test should still be valid LLVM IR that opt can run the pass on.

Step 2 — Author at least three tests

Place files at test/unit/<PassName>/0N_<short_name>.ll, numbered to match the existing convention (read what's already there).

Each test starts with a RUN: line and uses FileCheck directives. Template:

; RUN: opt -load-pass-plugin=%sclib -passes=<PassName> -S %s | FileCheck %s

; <one-line description of what this test exercises>

define i64 @<descriptive_name>(i64 %x) {
; CHECK-LABEL: @<descriptive_name>
; CHECK:       <expected post-transform pattern>
; CHECK-NOT:   <pattern that must be gone>
  ; ... pre-transform IR ...
  ret i64 %r
}

The required mix:

1. Positive synthesized test — minimal IR demonstrating the canonical transform. Use plain integer constants; avoid noise. CHECK both the new instruction and CHECK-NOT for the old one.
2. Negative synthesized test — IR the pass must leave alone (variable instead of constant, wrong type, attribute that disables the transform, edge case from the pass's guard conditions). CHECK that the original instruction is preserved and the new pattern is absent.
3. Benchmark-derived test (mandatory) — snippet from step 1, with a cost-delta comment block (see step 3). This file is the one that demonstrates "the pass is useful on real code."

If the pass has multiple distinct transformation rules (e.g. ShiftToMul: shl→mul, lshr→udiv, ashr→sdiv with sign guard), add at least one positive test per rule on top of the three-test minimum. The three-test floor is a floor, not a ceiling.

Step 3 — Annotate the benchmark-derived test with a TM cost delta

This is the part that's new and easy to skip. The benchmark-derived test must include a comment that:

• Names the source benchmark and where the snippet came from.
• Lists the before/after TM cost per executed instance.
• States whether the result depends on mul-fusion or 4-Phobia (call it out explicitly, don't bury it).
• Says heap is unchanged unless it isn't.

Use this format:

; Source: swpp202601-benchmarks/matmul1/src/matmul1.ll, inner loop body.
;
; TM cost (per executed instance):
;   before:  shl %v, 3        cost 10
;   after:   mul %v, 8         cost  2  (0 if fused with following add/sub)
;   delta:  -8 .. -10 per execution
;   4-Phobia: not triggered (no `4` in either op text)
;   heap:    unchanged

Refer to references/cost-cheatsheet.md (in this skill folder) for the per-op cost table, fusion rule, and 4-Phobia. For deeper questions consult .claude/skills/swpp-team-rules/references/target-machine-spec.md or the canonical PDF at Guide/Spec/SWPP202601 TM Specification.pdf.

If the cost delta is small or zero, say so and reconsider whether the pass is worth the maintenance — but still keep the test, because correctness coverage matters even when effectiveness is marginal.

Step 4 — Verify the tests run

Don't trust the test by inspection. Run them inside the CI image (ghcr.io/dennis0405/swpp202601-team03-ci:latest). Use the existing container if available:

# Preferred: exec into existing container (repo is already bind-mounted)
docker start <container_name>
docker exec <container_name> bash -lc '
  set -euo pipefail
  /work/swpp202601-team03/ci/scripts/1-build-compiler.sh
  /work/swpp202601-team03/ci/scripts/2-ctest-compiler.sh
'

# To run only the new pass tests:
docker exec <container_name> bash -lc '
  ctest --test-dir /work/swpp202601-team03/build -R <PassName> --output-on-failure
'

If no container exists:

docker run --rm \
  -v "$(git rev-parse --show-toplevel)":/work/swpp202601-team03 \
  -w /work/swpp202601-team03 \
  ghcr.io/dennis0405/swpp202601-team03-ci:latest \
  bash -lc '
    set -euo pipefail
    ci/scripts/1-build-compiler.sh
    ci/scripts/2-ctest-compiler.sh
  '

Step 5 — Draft the PR cost-analysis snippet

Hand the user a copy-pastable block for the PR description. It must repeat (in prose) the cost delta from the benchmark-derived test and reference the source line range. Example:

### Effectiveness

Benchmark-derived test `test/unit/ShiftToMul/03_matmul1_inner_loop.ll`
lifts the inner-loop body of `swpp202601-benchmarks/matmul1/src/matmul1.ll`.
Per iteration the pass replaces `shl %v, 3` (TM cost 10) with `mul %v, 8`
(TM cost 2, or 0 if fused with the trailing add). Loop runs `dim^3` times,
so on the standard `dim=64` input the saving is roughly 8 * 64^3 ≈ 2.1M
instruction-cost units, with no heap impact and no 4-Phobia trigger.

The numbers are illustrative; replace them with the actual values for the test you wrote.

Things to avoid

• Do not invent IR that the pass cannot actually fire on. The benchmark-derived test must come from a real benchmark file.
• Do not write the cost delta from memory — open references/cost-cheatsheet.md (or target-machine-spec.md) and walk the instructions.
• Do not skip the 4-Phobia check. Forgetting +10 on every 4-bearing instruction is the most common mis-quote of TM cost.
• Do not run cmake / clang / ctest on the host shell — only inside the CI image (/E2E-TEST or the docker run snippet above).
• Do not mass-edit unrelated existing tests "for consistency" — the rule is a floor of new coverage, not a re-style of the directory.
• Do not delete failing tests to make the suite green. Failure means the pass or the CHECK pattern is wrong — fix the underlying issue.

Reference files

• references/cost-cheatsheet.md — focused per-op cost table for unit-test annotations.
• Guide/Spec/SWPP202601 TM Specification.pdf — canonical TM spec (load when an edge case isn't in the cheatsheet).
• .claude/skills/swpp-team-rules/references/target-machine-spec.md — full markdown digest of the TM spec, already in this repo.
• Guide/Spec/UnitTestGuide.md — the team rule that this skill enforces.