bench-cost

name	bench-cost
description	Run a SWPP benchmark through the log-enabled interpreter and analyze cost breakdown. Use when asked to profile cost, debug performance, or understand where execution cost accumulates in a benchmark. Invoke as `/bench-cost [bench_name] [input_number]` — e.g. `/bench-cost matmul2 3`. With no args, runs all benchmarks without log analysis.

bench-cost — Benchmark Cost Analysis

Invocation

/bench-cost [bench_name] [input_number]

• bench_name — benchmark directory name, e.g. matmul2, bitcount1. Omit to run all.
• input_number — test case index, e.g. 3 for input3.txt. Omit to run all inputs for the chosen benchmark.

When both args are given, the log-enabled interpreter runs exactly one test case and the detailed log is analyzed.

Environment

All paths are container-internal. The CI image is:

ghcr.io/dennis0405/swpp202601-team03-ci:latest

Key paths:

Path	Content
/opt/interpreter/main	Normal interpreter (no log) — used by CI
/opt/interpreter/main-log	Log-enabled interpreter (--features=log) — use for cost debugging
/opt/benchmarks/	Pinned benchmark checkout with .ll and test I/O
/work/swpp202601-team03/	Compiler checkout (bind-mounted from host)

Log file location: main-log writes to the interpreter process's working directory at the time of execution. The path depends on how the container is launched:

Launch method	Log path (inside container)	Host-accessible?
docker exec <container> from workdir /work	/work/swpp-interpreter-basic.log	No (not bind-mounted)
docker exec <container> from workdir /work/swpp202601-team03	/work/swpp202601-team03/swpp-interpreter-basic.log	Yes
docker run --rm -w /work/swpp202601-team03	/work/swpp202601-team03/swpp-interpreter-basic.log	Yes

Read the log via docker exec <container> cat /work/swpp-interpreter-basic.log when it is not bind-mounted.

Procedure

Step 1: Find or start the container

First check for an existing container with the CI image:

docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" | grep swpp202601-team03-ci

If a stopped container exists (preferred — preserves state, no new pull needed):

docker start <container_name>
docker exec <container_name> bash -lc '
  INTERP=/opt/interpreter/main-log \
  BENCH_FILTER=<bench_name> \
  INPUT_FILTER=<input_number> \
  /work/swpp202601-team03/ci/scripts/5-interpreter.sh
'

Read the log from inside the container:

docker exec <container_name> cat /work/swpp-interpreter-basic.log

If no container exists (creates a clean container, log bind-mounted to host):

docker run --rm \
  -v /Users/junby/Coding/2026SWPP/swpp202601-team03:/work/swpp202601-team03 \
  -w /work/swpp202601-team03 \
  ghcr.io/dennis0405/swpp202601-team03-ci:latest \
  bash -lc '
    INTERP=/opt/interpreter/main-log \
    BENCH_FILTER=<bench_name> \
    INPUT_FILTER=<input_number> \
    ci/scripts/5-interpreter.sh
  '
# log appears on host at:
# /Users/junby/Coding/2026SWPP/swpp202601-team03/swpp-interpreter-basic.log

Replace <bench_name>, <input_number>, and <container_name> with actual values.

Note: docker run --rm creates a new temporary container and removes it on exit. It does NOT affect any existing containers.

Step 1a: Build compiler and generate assembly if needed

If .s files are missing (missing asm: error from step 1), build first:

docker exec <container_name> bash -lc '
  /work/swpp202601-team03/ci/scripts/1-build-compiler.sh
  /work/swpp202601-team03/ci/scripts/3-compiler.sh
'

Step 2: Read the log and cost output

After the run, collect:

1. Log file (docker exec <container> cat /work/swpp-interpreter-basic.log)
2. Cost file: /opt/test-result/<bench_name>/input<N>.cost — line: Final Cost : N
3. Cost summary: /opt/test-result/cost-summary.tsv

Compute heap cost: heap_cost = final_cost - instruction_cost. instruction_cost is the last TotalCost value in the log. peak_heap_bytes = heap_cost / 1024.

Step 3: Analyze cost breakdown with Python

Run this inside the container to get a per-instruction-type cost table:

docker exec <container_name> python3 - /work/swpp-interpreter-basic.log <<'PYEOF'
import sys
from collections import defaultdict

kind_cost = defaultdict(int)
kind_cnt = defaultdict(int)

with open(sys.argv[1]) as f:
    next(f)
    for line in f:
        parts = [p.strip() for p in line.split("|")]
        if len(parts) < 5:
            continue
        kind = parts[1]
        try:
            cost = int(parts[3])
        except:
            continue
        kind_cost[kind] += cost
        kind_cnt[kind] += 1

total = sum(kind_cost.values())
print("Total instruction cost: " + str(total))
print()
print("%-22s %8s %12s %7s" % ("Kind", "Count", "TotalCost", "pct"))
print("-" * 54)
for k, c in sorted(kind_cost.items(), key=lambda x: -x[1]):
    pct = c * 100.0 / total
    print("%-22s %8d %12d  %6.1f%%" % (k, kind_cnt[k], c, pct))
PYEOF

Step 4: Analyze cost breakdown

Decompose total cost:

Final Cost = instruction_cost + peak_heap_bytes × 1024

If heap cost dominates (>50%), free-earlier or stack-promotion optimizations yield the highest return.

Log column format:

Index | InstructionKind | LineNum | Cost | TotalCost | CurrentScope

Expensive patterns to flag:

Log entry	Why expensive	Optimization hint
Store cost >> 50	Heat: floor(HP/10) extra per access	Interleave non-memory instructions to cool; split malloc blocks
Load cost >> 50	Heat accumulation	Same as Store
Branch cost 90	Conditional true branch	Flip condition; move hot path to false-branch (cost 30)
UBranch cost 45	Forward unconditional jump (30×1.5)	Reorder blocks so this becomes backward
shl/lshr/ashr	Cost 10 vs mul cost 2	Replace shift with mul
Mul (not Mul-FMA) cost 2	FMA not triggered	Reorder: ensure add/sub immediately follows mul
EAdd cost 20	Parity prediction wrong	Improve parity analysis or use plain add
Malloc/Free in hot path	150 each	Hoist out of loops; reuse buffers

Heat analysis:

• Segments accumulate 400 HP per access, capped at 2000 HP
• Extra cost = floor(HP / 10) per access (max +200)
• Every non-access instruction cools by its own cost
• Different malloc blocks are heat-independent even if adjacent

Forward-jump penalty:

• Any jump target that appears later in the code: cost × 1.5
• br false/unconditional: 30 → 45 if forward
• br true: 90 → 135 if forward
• Put loop headers before loop bodies so back-edges are backward (cheap)

Step 5: Report findings

Summarize:

1. Final cost = instruction cost + heap cost (show the split)
2. Top 3–5 most expensive instruction types from the table
3. Specific opportunities with estimated savings (e.g. "flipping N branches saves N×60")
4. Any correctness failures (diff output)