<div align="center">
# CAE-bench
CAE-bench is SWE-bench for engineering simulation. It measures whether a model can do computer-aided engineering — FEA, CFD, multibody dynamics, durability and optimization — in real CAE software. Tasks come from real engineering practice, not synthesized from papers. The current release covers CFD, combustion, battery and electronics-packaging thermal tasks.
> **An industrial simulation agent benchmark.** Hand an LLM agent a real
> CAE task — meshing, boundary conditions, solver invocation, log parsing,
> KPI extraction — and grade what it actually produced. No LLM-as-judge: the
> verifier re-runs the agent's own extraction commands against the solver
> artifacts the agent left on disk, and checks the numbers against a tolerance
> band that can be argued for.
<p align="center">
<a href="#quick-start"><img src="https://img.shields.io/badge/Quick_Start-2_min-3b82f6?style=for-the-badge" alt="Quick Start"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-eab308?style=for-the-badge" alt="License"></a>
</p>
<p align="center">
<img src="https://img.shields.io/badge/runner-Harbor-009688" alt="Harbor">
<img src="https://img.shields.io/badge/scope-CAE_simulation-blue" alt="Scope">
<img src="https://img.shields.io/badge/judge-code,_not_LLM-7c3aed" alt="Code judge">
</p>
[What this measures](#what-this-measures) · [Scope](#scope) · [Scoring](#how-scoring-works-in-60-seconds) · [Reference runs](#reference-runs) · [Quick start](#quick-start) · [Layout](#repository-layout)
</div>
---
## What this measures
Every existing engineering-agent benchmark scores *"does the code run / does
the testbench pass / is the file valid."* CAE-bench scores the empty quadrant:
**did the agent drive the simulation correctly** — a numeric result inside a
tolerance band whose placement is defensible, rather than a percentage someone
picked — **across many solvers driven through one interface.**
Note what that is *not*. This is not a V&V exercise and the score does not ask
whether the model reproduces reality: coarse-mesh RANS sitting 20% off a wind
tunnel is a fact about RANS, not about the agent. The reference is what a
competent practitioner gets with this class of tool at this operating point —
published cross-code results at the same grid level where they exist, a
calibrated oracle where they do not. Experimental data is provenance and sanity
check, not the scoring basis. What V&V reasoning contributes is narrower and
borrowed: where the band sits, and how narrow it may get before it is scoring
noise.
Each task hands the agent a natural-language problem, a container with a real
solver installed, and one rule: produce `/tmp/agent/result.json` whose KPIs
carry **source provenance** — `(value, source.kind, source.path,
source.extract)`. The verifier opens the named artifact, re-runs the declared
extraction, and confirms the value. Hand-written numbers, fabricated logs, and
unreproducible KPIs score zero. A reported value with **no solver artifact on
disk** scores zero regardless of what stdout claimed — this is what stops the
"recall the closed-form answer and skip the solver" shortcut.
**Any launch route, one measurement.** The score is the computed result,
checked by code. The agent drives the solver however it likes — `sim-cli`,
native solver CLIs, Python wrappers or its own scratch scripts are all valid
launch routes — because tooling is implementation and the engineering result is
what is measured.
## Scope
CAE-bench measures how well LLM agents do computer-aided engineering, in the
sense [Wikipedia](https://en.wikipedia.org/wiki/Computer-aided_engineering)
gives it: finite element analysis (FEA), computational fluid dynamics (CFD),
multibody dynamics (MBD), durability and optimization, done in real CAE
software. The current release covers **CFD, combustion, battery and 2.5D
electronics-packaging thermal and warpage** tasks.
Built on [Harbor](https://www.harborframework.com/): each task is a
Harbor-compatible directory (Dockerfile + instruction + `solve.sh` oracle +
`test.sh` verifier). We produce the tasks and the board — published at
[cae-bench.svdailab.com](https://cae-bench.svdailab.com) — and Harbor's runner
and registry do the rest.
Cases are organized by **work domain**, not by solver — one domain can host
several solvers and substrates. Most domains run on Harbor; the license-locked
multi-physics suites (COMSOL, Simulink) run **Path B** — Windows-native,
private, *not on Harbor*.
Each task ships a deterministic `solution/solve.sh` oracle (we require it,
where Harbor makes it optional) that produces the verifier's upper-bound sanity
check.
Which domains exist, which solver backs each, and how ready each one is are all
derived rather than restated here — no inventory in this file to fall out of
date:
- **`CASES.md`** — every case with a derived state (ready / legacy / draft /
blocked) and a per-track rollup. **Not committed here** (`.gitignore`): a
clone reads the cases themselves, and a rendered catalog is only needed by the
public mirror. Render your own with `python tools/gen_cases_md.py`.
- **[`environment/domains/README.md`](environment/domains/README.md)** +
**[`VERSIONS.md`](environment/domains/VERSIONS.md)** — the domain images,
substrates and toolchain versions.
- **[`SCHEMA.md`](SCHEMA.md)** — the case/verifier contract.
## How scoring works in 60 seconds
Every case declares named KPIs and how to measure them in `tests/kpis.json`.
The agent submits each KPI with the command that produced it:
```json
{
"pressure_drop": {
"value": 42.0,
"source": { "kind": "file_extract",
"path": "/root/case/postProcessing/pressureDrop/0/fieldValue.dat",
"extract": "tail -1 /root/case/postProcessing/pressureDrop/0/fieldValue.dat | awk '{print $2}'" }
}
}
```
The verifier opens `path`, runs the declared extraction, and compares the
result against ground truth within the declared tolerance. Per KPI:
```
kpi_score = source_verified · physics_pass · band_pass
source_verified — the agent's value re-extracts from the file it named
physics_pass — the value lies inside [physics_min, physics_max]
band_pass — BINARY: |value − gt_value| ≤ pass_tol → 1, else 0
```
`band_pass` has no partial credit, deliberately: a computed number that is
wrong is not less wrong for being wrong by less. The full contract, and the one
place this chain is stated normatively, is [`SCHEMA.md`](SCHEMA.md).
A per-solver **artifact detector** (`openfoam` → `polyMesh/` + time dirs,
`calculix` → `.frd`/`.dat`, `cantera` and `pybamm` → solver state dumps or
logs, …) hard-zeros the aggregate if the
solver left no evidence of a real run. The case-level score is a weighted sum
of per-group means; templates (`measurement` / `numerical` / `workflow`) set
the group weights. Full contract in [`SCHEMA.md`](SCHEMA.md).
## Reference runs
Scores are not kept in this repo. A leaderboard row is one model, on one
commit, in one environment — cheap to recompute from the stored artifacts and
stale the moment a toolchain moves. The published numbers live on the board at
[cae-bench.svdailab.com](https://cae-bench.svdailab.com);
[`LEADERBOARD.md`](LEADERBOARD.md) holds the ranking method and the accounting
rules that make two runs comparable; [`ORACLE.md`](ORACLE.md) is how you check
your own environment first.
What the runs so far show is workflow-shaped rather than numeric:
artifact-grounded grading separates agents that can *talk about* simulation
from agents that can produce solver artifacts surviving replay. CFD is the
harder suite (author case files → mesh → solve → post-process fields →
replayable provenance).
## Quick start
```bash
# 1. install harbor (the runner — same one Terminal-Bench uses)
uv tool install harbor
# 2. clone
git clone https://github.com/svd-ai-lab/cae-bench && cd cae-bench
# 3. oracle smoke on one packaging case (no LLM, no API key)
harbor run -p cases/packaging/thermal -i pkg25d_thermal_tim118um_k4p7_h22600 --agent oracle -y
# 4. oracle smoke on a CFD case (also no LLM; needs the cfd base
# image — see REPRODUCING.md for local builds)
harbor run -p cases/cfd/fluids -i taylor_green_vortex_2d_re100 --agent oracle -y
```
The oracle involves no LLM and no sampling, so the same case on the same image
gives the same number every time. That number is not always 1.000: a case scored
against published experimental or cross-code data sits below 1.0 by design,
while an oracle-self-calibrated case sits at 1.0. What matters is that **your
number matches the one the oracle produces elsewhere** — if it doesn't, the bug
is in your environment, not the agent. See [`ORACLE.md`](ORACLE.md). To run a model instead of the oracle, swap
`--agent oracle` for `--agent terminus-2 --model <model-id>` (or your own
wrapper). `terminus-2` is the scaffold the published board runs on, so a row
produced any other way is a diagnostic rather than a board cell. The three
reproduction paths (GHCR pull / build from source / paranoid) are in
[`REPRODUCING.md`](REPRODUCING.md); what happens inside a trial is Harbor's,
and the launcher that drives it is
[`tools/run_harbor_trial.sh`](tools/run_harbor_trial.sh).
## Repository layout
```text
cae-bench/
├── cases/<domain>/<subdomain>/<case-id>/ # domain → physics-class → case
│ ├── task.toml # Harbor schema 1.3 + [metadata.sim]
│ ├── instruction.md # prompt handed verbatim to the agent
│ ├── solution/solve.sh # deterministic oracle (required)
│ └── tests/{kpis.json,test.sh} # neutral-v0.3 KPIs + verifier entry
├── environment/
│ ├── domains/ # one fullstack image per domain (+ VERSIONS.md)
│ └── base/ # shared substrate
├── lib/sim_benchmark_verifier/ # the grader + per-solver detectors (Python)
├── tools/ # harness, lint, oracle runners, aggregation
├── docs/ # methodology, competitive landscape, demand records
├── results/ # per-run record artifacts
├── SCHEMA.md · LEADERBOARD.md · ORACLE.md · REPRODUCING.md
│ # (CASES.md is generated, not committed)
```
## Contributing
Issue-first: a case PR without a linked, approved proposal issue gets closed
(keeps the suite from becoming a drive-by case dump). Two common contributions:
- **A new case.** Copy a sibling case in the same domain as a template (e.g.
under [`cases/combustion/`](cases/combustion/) or
[`cases/cfd/fluids/`](cases/cfd/fluids/)); fill
`task.toml` + `instruction.md` + `tests/kpis.json` + `solution/solve.sh`.
`solve.sh` must score exactly 1.0 (scoring is binary against the band, so
a fraction means a whole KPI group did not pass) and a deliberately-broken
run must score < 0.5. The contract is [`SCHEMA.md`](SCHEMA.md).
- **A model.** Harbor owns the runner, so a new model is a credential set and
an agent name, not a harness — see
[`tools/run_harbor_trial.sh`](tools/run_harbor_trial.sh). Putting that model
on the **published board** is a further four steps (label, population,
scaffold, snapshot), and the launcher's defaults are not the board's: the
procedure is `docs/internal/running_the_board.md`.
Bringing a brand-new solver onto the track follows contract → oracle → runtime;
see [`.claude/skills/solver-benchmark-blueprint`](.claude/skills/solver-benchmark-blueprint/SKILL.md).
## Citing
```bibtex
@misc{caebench2026,
title = {CAE-bench: An Industrial Simulation Agent Benchmark},
author = {{svd-ai-lab}},
year = {2026},
url = {https://github.com/svd-ai-lab/cae-bench}
}
```
## License
Apache 2.0. See [`LICENSE`](LICENSE). Bundled example assets (meshes, grids,
input decks) remain under their respective upstream licenses; see each case's
`solution/` directory for source attribution.