| name | vtestrun |
| description | Run all IP-level testcases from tc_list.md through the bench Makefile, capture results, and write per-failure issue reports with no fix suggestions |
| effort | medium |
| model | sonnet |
Verilog Testcase Runner
Overview
This skill runs every IP-level testcase listed in tc_list.md through the
Makefile-driven bench that vtestgen scaffolded, captures the build/simulation
logs, and writes a structured issue report for each failure.
The input is an IP name. The outputs are:
./issue/<ip>/issue_NNN_<tcname>.md — one file per failing testcase./issue/<ip>/run_summary.md — overall pass/fail count and failure index
This skill reports failure context only. It does not analyze root cause, propose
fixes, or suggest code changes.
Prerequisites
| Tool | Purpose | Check |
|---|
| xvlog | compile / lint | xvlog --version |
| xelab | elaborate | xelab --version |
| xsim | simulate | xsim --version |
| make | build orchestration | make --version |
Step 1 - Read the testcase list
Read tb/<ip>/tc_list.md and parse the testcase table into a run list. Include only
rows with status OK; skip rows with status SKIPPED.
For each entry record: file (the .sv), module (the tb_* module, equal to the
file base name), category, and description. Confirm tb/<ip>/Makefile and
tb/<ip>/script/rtl.f exist before proceeding.
Step 2 - Create the output directory
Create ./issue/<ip>/ if absent. Do not delete existing issue files; new files use
the next available NNN index (continue from the highest existing index, else start
at 001).
Step 3 - Run each testcase
Run from inside tb/<ip>/. vtestrun is the authoritative pass/fail, but it
does not need to re-simulate testcases that vtestgen already ran — the
verdict is the same [FINISH] token. Re-running everything would just repeat
vtestgen's self-check sims. The one thing it must never do is trust a stale
result (a simulate.log older than an RTL/TB source it depends on — e.g. after a
Phase 6 RTL fix), because the build rule no-ops while a simulate.log exists and
make cleanf keeps prior PASS dirs.
So reuse a fresh result, rebuild only a stale (or missing) one — per testcase:
if [ -f sim/tb_<module>/simulate.log ] && \
[ -z "$(find ../../rtl ../../lib *.vh sti_*.sv sco_*.sv tb_<module>.sv \
-newer sim/tb_<module>/simulate.log 2>/dev/null)" ]; then
:
else
rm -rf sim/tb_<module> && make tb_<module>
fi
This makes the first run right after vtestgen a pure verdict-collection pass
(no second simulation), while a post-fix re-test rebuilds every testcase whose
RTL/TB changed (a fix touches ../../rtl, so all dependent testcases re-run and a
regression is still caught). If you ever want to force the whole suite fresh
regardless of mtimes, use make retest (= clean all).
make tb_<module> produces sim/tb_<module>/ containing compile.log,
elaborate.log, and simulate.log. After all testcases, make report prints a
one-line PASS/FAIL per testcase; use it to cross-check.
Pass/fail decision
The decision is based solely on the [FINISH] verdict token in
simulate.log (see .claude/skills/shared/CodingStyle.md). Do not decide from the
presence of [ERROR] text — [ERROR] lines are collected only for the issue
report, not for the verdict.
A testcase is a FAIL if any of the following is true:
compile.log (xvlog) or elaborate.log (xelab) reports an error / the build step failedsimulate.log contains [FINISH] FAILsimulate.log contains no [FINISH] line at all (crash, or the run was killed by the timeout)
A testcase is a PASS only if simulate.log contains [FINISH] PASS.
Step 4 - Write issue reports
For each FAIL, assign the next sequential issue number (zero-padded to 3 digits) and
write ./issue/<ip>/issue_NNN_<module>.md. Fill every section from the captured logs.
Do not add analysis, speculation, or fix suggestions.
Section: Testcase
A table: File (tb/<ip>/<file>), Module, Category, Description.
Section: Failure Summary
The verbatim first [ERROR] line from simulate.log (the most diagnostic message).
If there is no [ERROR] line — e.g. timeout or crash — use the [FINISH] FAIL line,
or the first error line from compile.log/elaborate.log if the build failed. One
line only.
Section: Simulation Log
A fenced code block containing, in order of occurrence, up to the first 10 [ERROR]
lines from simulate.log and the [FINISH] FAIL line. If the build failed, include
the xvlog/xelab error lines from compile.log/elaborate.log instead and leave the
simulation portion empty. If the testcase timed out, include the last 10 lines of
simulate.log before termination.
Section: Reproduction
A fenced bash block:
cd tb/<ip>
make cleanf
make tb_<module>
Closing note
End the file with: > This report documents observed failure context only. No fix is proposed.
Rules for the Simulation Log section:
- Include up to the first 10
[ERROR] lines produced by cpu_read_check or explicit checks.
- Do not include
[INFO] lines or Vivado tool banners.
- For a build failure, include the full xvlog/xelab error output.
Step 5 - Write the run summary
After all testcases run, write ./issue/<ip>/run_summary.md. Keep the Results
table as the one-metric-per-row shape below: vflow's scanner (scan.sh) reads
the Failed count from the | Failed | N | row to decide whether Phase 6 is
needed, so the count must stay in the second column of that row.
# Test Run Summary: <ip>
## Results
| Metric | Count |
|--------------------|-------|
| Total run | N |
| Passed | N |
| Failed | N |
| Skipped (vtestgen) | N |
## Failures
| Issue | Testcase | Category | Failure Summary |
|-------|----------|----------|-----------------|
| [001](issue_001_<module>.md) | tb_<name> | <category> | <first [ERROR] or [FINISH] FAIL line, truncated to 80 chars> |
## Skipped Testcases
| Testcase | Reason |
|----------|--------|
| tb_<name> | Skipped self-check in vtestgen |
---
> Run completed: <date and time>
If all testcases passed, write the summary with zero failures and omit the Failures
table.
Known Issues and Fixes
