llvm-miscompile-reduce

name	llvm-miscompile-reduce
description	Reduce LLVM miscompilation reproducers — LLUBI/Alive2 oracle + opt-bisect-limit + llvm-reduce

Tools

All LLVM tools are on PATH: opt, llc, lli, llvm-reduce, clang, alive-tv, llubi_legacy, llvm-extract.

Timeout rule: wrap every standalone opt, llc, lli, or clang command with timeout 60. llubi_legacy --reduce-mode --max-steps 1000000 is sufficient. interestingness.sh commands already carry timeouts — no extra wrapping needed there.

Miscompilation Reduction Pipeline

CRITICAL: Reduction operates exclusively on LLVM IR. Never compile IR to native binaries for verification — use the oracle tools (llubi_legacy, alive-tv, lli) directly on IR.

0. Read metadata from extract.json

Read extract.json and note:

• oracle — opt for middle-end, llc for backend.
• args — the opt/llc/lli arguments. For oracle=opt this is the opt pipeline (e.g. -passes='default<O2>'). For oracle=llc this is the llc/lli args (usually "" for backend miscompilation — the reproducer IR is already fully optimized by clang).
• reproducer_file — the .ll file to reduce. For backend miscompilation with oracle=llc, this is full_opt.ll (already optimized — no bisect needed).
• pattern — how the miscompilation manifests: wrong_output, nonzero_exit, or infinite_loop. The interestingness script MUST be written to preserve this exact pattern type — do NOT change wrong_output into a crash check or vice versa.

Create a symlink for convenience:

ln -sf <reproducer_file> repro.ll

1. Choose bisect/reduce oracle

Based on extract.json oracle:

• oracle=opt (middle-end) → use llubi_legacy for bisect and reduce
• oracle=llc (backend) → use lli for reduce (no bisect needed — the reproducer IR from clang is already fully optimized)

CRITICAL — lli preprocessing: Before using the lli oracle, preprocess the IR to remove main() argument dependencies. If main() uses argc/argv, strip those references from the IR (e.g., replace argc with a constant). Without this, llubi_legacy and lli may produce different output even on a correct backend because llubi_legacy does not pass command-line arguments.

2. Reproduce the miscompilation

Middle-end (llubi):

set -o pipefail
timeout 60 llubi_legacy --reduce-mode --max-steps 1000000 repro.ll > ref_ubi
! opt -passes='<args>' repro.ll -S | llubi_legacy --reduce-mode --max-steps 1000000 - | diff -q ref_ubi -

Backend (lli — no bisect, IR is already optimized):

set -e
timeout 60 llubi_legacy --reduce-mode --max-steps 1000000 repro.ll > ref_ubi
timeout 10 lli <args> repro.ll > _lli_out
! diff -q ref_ubi _lli_out

ACCEPTED RISK: Crashes in the pipeline (opt, llubi_legacy, or lli segfault) are treated as miscompilation: pipefail makes the pipeline exit non-zero on crash, ! inverts that to exit 0 ("miscompilation found"). The daemon's final verify() step independently checks the reduced IR and will reject cases where the miscompilation does not actually reproduce, so a crash-confused reduction is caught at verification time.

3. opt-bisect-limit binary search to find single pass

Middle-end only (oracle=opt). For backend miscompilation (oracle=llc), skip to step 5 — the reproducer IR is already optimized and no bisect is needed.

First, pre-compute the reference output and get total pass count:

timeout 60 llubi_legacy --reduce-mode --max-steps 1000000 repro.ll > ref_ubi
timeout 60 opt -opt-bisect-limit=-1 -passes='<args>' repro.ll -S -o /dev/null 2>&1   → total=N

Write a bisect script — do NOT run the binary search inline (llvm-reduce style):

Mid-end (llubi oracle):

cat > bisect.sh <<'SCRIPT'
#!/bin/bash
set -e
M="$1"
ref="$2"
ir="$3"
timeout 30 opt -opt-bisect-limit="$M" -passes='<args>' "$ir" -S > _bisect_opt.ll
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 _bisect_opt.ll > _bisect_out.txt
! diff -q "$ref" _bisect_out.txt
SCRIPT
chmod +x bisect.sh

Then binary search: lo=1, hi=N. At each step run bisect.sh M ref_ubi repro.ll:

• exit 0 → miscompilation at or before M → hi=M
• exit 1 → correct up to M → lo=M+1

Backend (lli oracle):

cat > bisect.sh <<'SCRIPT'
#!/bin/bash
set -e
M="$1"
ref="$2"
ir="$3"
timeout 30 opt -opt-bisect-limit="$M" -passes='<args>' "$ir" -S > _bisect_opt.ll
timeout 10 lli _bisect_opt.ll > _bisect_out.txt
! diff -q "$ref" _bisect_out.txt
SCRIPT
chmod +x bisect.sh

Then binary search, same as above.

ACCEPTED RISK: Crash → miscompilation. set -e causes the script to exit non-zero if opt crashes, and ! diff -q inverts: if oracle crashes, diff exits non-zero (ref exists, _bisect_out.txt missing/empty), ! returns 0. The daemon's verify() step independently confirms.

IMPORTANT: diff -q only compares exit code (0=same, 1=differ), no content output. set -e exits early if opt fails (no _bisect_opt.ll). The reference output is computed once, not inside the loop. Each bisect step uses temp files, avoiding pipefail complexity.

4. Extract the single pass name and capture IR before it

The bisect log prints the last pass run before the miscompilation (e.g. BISECT: running pass (N) GVN on ...). Convert this to the -passes= form (e.g. -passes=gvn). Do NOT guess from filenames.

Capture the IR just before the bad pass:

opt -opt-bisect-limit=M-1 -passes='<args>' repro.ll -S > before.ll

5. llvm-reduce with ONLY the single pass

CRITICAL: The interestingness script MUST match the pattern from extract.json. Choose the template for the pattern type. Preserving the exact pattern type ensures the reduced IR manifests the same kind of miscompilation — wrong_output stays wrong_output, nonzero_exit stays nonzero_exit, infinite_loop stays infinite_loop.

All miscompilation interestingness scripts MUST also reject IR containing undef — undef masks genuine miscompilations. Add if grep -q " undef" "$1"; then exit 1; fi as the first check in every template below.

All interestingness scripts MUST also reject IR with target intrinsics but no target-features — llvm-reduce will strip the attribute otherwise, breaking oracle commands. Add the following guard after the undef check in every template below:

if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi

llubi oracle (middle-end) — pattern=wrong_output:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -eo pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt
timeout 30 opt -passes='<pass_name>' "$1" -S > _opt.ll
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 _opt.ll > _out.txt
! diff -q _ref.txt _out.txt
SCRIPT

llubi oracle (middle-end) — pattern=nonzero_exit:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -o pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt || exit 1
timeout 30 opt -passes='<pass_name>' "$1" -S | timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 -
ret=$?
# Exit 0 (interesting) if pipeline failed with crash/signal/assert — NOT timeout (124)
test $ret -ne 0 -a $ret -ne 124
SCRIPT

llubi oracle (middle-end) — pattern=infinite_loop:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -o pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt || exit 1
timeout 30 opt -passes='<pass_name>' "$1" -S | timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 -
ret=$?
# Exit 0 (interesting) only if pipeline timed out
test $ret -eq 124
SCRIPT

lli oracle (backend) — pattern=wrong_output:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -eo pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
# Reject IR where main() has parameters — llubi_legacy and lli disagree on argc/argv
grep -qP 'define\s+\S+\s+@main\s*\(\s*\)' "$1" || exit 1
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt
timeout 10 lli <args> "$1" > _out.txt
! diff -q _ref.txt _out.txt
SCRIPT

lli oracle (backend) — pattern=nonzero_exit:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -o pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
# Reject IR where main() has parameters — llubi_legacy and lli disagree on argc/argv
grep -qP 'define\s+\S+\s+@main\s*\(\s*\)' "$1" || exit 1
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt || exit 1
timeout 10 lli <args> "$1" > /dev/null
ret=$?
# Exit 0 (interesting) if pipeline failed with crash/signal/assert — NOT timeout (124)
test $ret -ne 0 -a $ret -ne 124
SCRIPT

lli oracle (backend) — pattern=infinite_loop:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -o pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
# Reject IR where main() has parameters — llubi_legacy and lli disagree on argc/argv
grep -qP 'define\s+\S+\s+@main\s*\(\s*\)' "$1" || exit 1
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt || exit 1
timeout 10 lli <args> "$1" > /dev/null
ret=$?
# Exit 0 (interesting) only if pipeline timed out
test $ret -eq 124
SCRIPT

lli oracle (backend) — pattern=nonzero_exit:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -o pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt || exit 1
timeout 30 opt -passes='<pass_name>' "$1" -S | timeout 120 lli -
ret=$?
# Exit 0 (interesting) if pipeline failed with crash/signal/assert — NOT timeout (124)
test $ret -ne 0 -a $ret -ne 124
SCRIPT

lli oracle (backend) — pattern=infinite_loop:

cat > interestingness.sh <<'SCRIPT'
#!/bin/bash
set -o pipefail
if grep -q " undef" "$1"; then exit 1; fi
if grep -qP 'declare.*@llvm\.(x86|aarch64|arm|nvptx|amdgcn)\.' "$1"; then
  grep -q 'target-features' "$1" || exit 1
fi
timeout 120 llubi_legacy --reduce-mode --max-steps 1000000 "$1" > _ref.txt || exit 1
timeout 30 opt -passes='<pass_name>' "$1" -S | timeout 120 lli -
ret=$?
# Exit 0 (interesting) only if pipeline timed out
test $ret -eq 124
SCRIPT

Then:

chmod +x interestingness.sh
llvm-reduce --test=interestingness.sh before.ll

Output: reduced.ll

If llvm-reduce gets stuck on a specific delta pass (check its progress output for a pass that keeps running without making progress), kill it and retry with --skip-delta-passes=<pass_name> (e.g. --skip-delta-passes=instructions). Repeat if it gets stuck on another pass.

6. Write checkpoint result (REQUIRED)

CRITICAL: After llvm-reduce produces a working reduced.ll, write result.json IMMEDIATELY. This saves a valid result before attempting optional oracle upgrades and manual reduction. The daemon accepts this as a completed reduction even if manual steps run out of time.

Middle-end (llubi):

{
  "type": "miscompilation",
  "args": "-passes=<pass_name>",
  "ir_file": "reduced.ll",
  "reference_file": "repro.ll",
  "oracle": "llubi",
  "llubi_args": "--reduce-mode --max-steps 1000000",
  "alive2_args": ""
}

Backend (lli):

{
  "type": "miscompilation",
  "args": "<lli_args from extract.json>",
  "ir_file": "reduced.ll",
  "reference_file": "repro.ll",
  "oracle": "lli",
  "llubi_args": "--reduce-mode --max-steps 1000000",
  "lli_args": ""
}

7. Try alive2 upgrade (middle-end only, optional)

For middle-end bugs with a function pass, try upgrading the oracle from llubi to alive2. This produces a stronger result.

alive2 single-function requirement: The IR submitted to alive2 MUST contain exactly 1 function definition (other declare declarations and global variables are allowed as needed). The verification step will reject IR with 0 or 2+ function definitions. Use llvm-extract to isolate a single function before running alive-tv.

Determine if the buggy pass is a function pass. If YES, extract a single function:

llvm-extract -func=<function_name> before.ll -S -o single_func.ll

Verify the extracted file contains exactly 1 define:

grep -c '^define ' single_func.ll

Output must be 1. If llvm-extract produces 0 or 2+ definitions (unusual), manually edit the IR to keep only the buggy function's definition (preserve any needed declare declarations and global variables).

Test with alive-tv:

opt -passes='<pass_name>' single_func.ll -S > __opt.ll
alive-tv --disable-undef-input --smt-to=10000 single_func.ll __opt.ll

Check the output:

• "incorrect transformation" count > 0 or "ERROR: Value mismatch" → alive2 upgrade succeeded. Before writing result.json, verify single_func.ll has exactly 1 function definition. Then update result.json with oracle: "alive2", alive2_args: "--disable-undef-input --smt-to=10000", llubi_args: "", and set ir_file to the single-function .ll file.
• "0 incorrect transformations" + "Transformation seems to be correct!" → no bug visible to alive2, keep llubi.
• "Alive2 approximated the semantics" → NOT a valid upgrade, keep llubi.
• Unsupported intrinsic/metadata/function → NOT a valid upgrade, keep llubi.

Only attempt alive2 for function pass bugs. For module pass bugs (e.g. inliner, IPSCCP, globalopt), skip this step.

8. Additional manual reduction (optional — only if time permits)

After the checkpoint result.json, try these techniques to shrink reduced.ll further. Test after each change that the miscompilation still reproduces. If any succeeds, update result.json with the improved ir_file.

Reduce bitwidth: Replace i64 with smaller integer types (i32, i16, i8) where possible. Adjust constants accordingly. Test that the miscompilation still reproduces.

Reduce pointer width: In the target datalayout, change pointer size to p:8:8 (or appropriate small size for the target).

Reduce loop trip count: If the IR has a loop with a fixed trip count (e.g. br i1 %cmp, label %loop, label %exit where %cmp compares induction variable against a constant like 128), reduce the constant (e.g. 128 → 4). This shrinks the loop body that needs to be preserved.

Loop transformations (alive2): For bugs involving loop passes, use alive-tv's loop unrolling flags to help it reason about loops:

alive-tv --disable-undef-input --smt-to=10000 -src-unroll=4 -tgt-unroll=4 <src> <tgt>

This unrolls loops in both source and target up to N iterations.

NEVER use undef. The reduced IR MUST NOT contain undef values — they cause non-deterministic behavior and can mask real miscompilations across all oracles (llubi, alive2, lli). If the original reproducer contains undef, replace it with zeroinitializer (for aggregates), null (for pointers), or explicit constant values (e.g. i32 0). The interestingness script and verification step will reject IR that still contains undef.

Strip fast math flags. If the IR contains fast or other fast-math flags on floating-point instructions, decompose fast into its constituent flags and keep only nnan and ninf — rewrite fast as nnan ninf explicitly. For any other fast-math flags (nsz, arcp, contract, afn, reassoc), remove them. If the miscompilation is specifically related to nsz (no-signed-zeros), prefer to drop nsz entirely rather than preserve it.

9. Verify final result

Verify the reduced IR still reproduces the miscompilation with the single pass. Write the final result.json (update from checkpoint if alive2 upgrade or manual reduction succeeded).

args field requirements: After bisect isolates the bug to a single pass (or a few specific passes), args MUST include that pass (e.g. -passes=gvn). Auxiliary flags that help reproduce the bug (e.g. -slp-threshold=-99999) may be included alongside the pass when relevant. The args field MUST NOT contain -opt-bisect-limit (bisect is a diagnostic step, NOT stored in result.json) and MUST NOT contain default< (the full O1/O2/O3 pipeline — bisect already narrowed it to the specific problematic pass). Backend/codegen passes MUST use legacy PM: when invoking backend passes like codegenprepare with opt, use -codegenprepare (legacy syntax), never -passes=codegenprepare (the new pass manager does not register codegen passes).

result.json (alive2):

{
  "type": "miscompilation",
  "args": "-passes=gvn",
  "ir_file": "reduced.ll",
  "reference_file": "repro.ll",
  "oracle": "alive2",
  "llubi_args": "",
  "alive2_args": "--disable-undef-input --smt-to=10000"
}

The ir_file for alive2 oracle MUST contain exactly 1 function definition. Function declarations (declare) and global variables are unrestricted. The verification step will reject IR with 0 or 2+ function definitions.

result.json (llubi):

{
  "type": "miscompilation",
  "args": "-passes=gvn",
  "ir_file": "reduced.ll",
  "reference_file": "repro.ll",
  "oracle": "llubi",
  "llubi_args": "--reduce-mode --max-steps 1000000",
  "alive2_args": ""
}

result.json (lli — backend):

{
  "type": "miscompilation",
  "args": "<lli_args from extract.json>",
  "ir_file": "reduced.ll",
  "reference_file": "repro.ll",
  "oracle": "lli",
  "llubi_args": "--reduce-mode --max-steps 1000000",
  "lli_args": ""
}

Error handling

• Oracle crash on original IR: report in error field
• If bisect cannot isolate a single pass: report the smallest pipeline possible in args
• If alive2 reports approximation or unsupported intrinsics: keep llubi, do NOT upgrade
• If all reduction attempts fail, write result.json with the FULL schema plus an error field describing the reason. The daemon requires all schema fields to be present — a bare {"error": "..."} will fail validation. Use:

{
  "type": "miscompilation",
  "args": "",
  "ir_file": "error.ll",
  "reference_file": "repro.ll",
  "oracle": "llubi",
  "llubi_args": "--reduce-mode --max-steps 1000000",
  "alive2_args": "",
  "error": "brief description of what failed"
}

• Do NOT generate a report.md file — the daemon handles report generation
• CRITICAL: All files stay in current working directory, never /tmp, /home, /etc, /var, or any other system path