# aspire-bench
A local [Vally](https://microsoft.github.io/vally/) harness comparing **agent
effectiveness** on a raw application and its aspirified counterpart. This is not
an HTTP load test. The scenario is **investigate startup failures, repair,
launch and verify the application, then assess services, database and Redis health**:
objective success, elapsed agent time, tokens, tool calls, and turns.
Bingo is a licensed snapshot of
[aspireify-workshop's `sebros/bench` branch](https://github.com/sebastienros/aspireify-workshop/tree/sebros/bench).
Both fixtures contain the same C#/SignalR backend, migration worker, data
model, and Vue/Vite frontend, with PostgreSQL 18.3 and Redis 8.6. The treatment
uses `03-observe/csharp` with ServiceDefaults and OTel.
Exact commits, licenses, and adaptations are recorded in
[`apps/bingo/provenance.json`](apps/bingo/provenance.json) and
[`treatment/provenance.json`](treatment/provenance.json).
The independent [MSBench / Harbor-native benchmark](msbench/README.md) contains
four ready-configured task folders: three raw guidance flavors and the full
Aspire skills/MCP treatment. Select cells using standard
`msbench-cli run --variant` commands; use Harbor's oracle mode for local image checks without
model inference. The two benchmark structures are maintained separately;
there is no conversion or synchronization.
## Why this harness exists
The benchmark measures how effectively an agent starts and repairs a real
application, then assesses its services, database and Redis health. Raw and
Aspire variants share the task, model, limits and live objective checks.
README guidance, Aspire skills/MCP and injected startup faults are explicit
experimental factors. We compare success first, then elapsed agent time,
tokens, tool calls and turns; this is not a load test or a sandbox benchmark.
Vally supplies the evaluation engine. This repository adds the application
fixtures, controlled treatments and application-specific lifecycle/proof:
| Responsibility | Vally | This harness on the local backend |
|---|---|---|
| Experiment and execution | Native variant resolution, file/skill staging, comma-separated model matrix, sequential conversation turns and inference through the SDK executor | Selects/repeats variant batches, supplies owned runtime context and delegates execution to the native executor |
| Grading and results | Per-turn diff graders, JSON `custom-metrics`, program-grader execution, metrics, trajectories, reports and `serve` dashboard | Validates endpoints and independently exercises the live application; records cleanup-aware verdicts and exports named variants/models for the dashboard |
| Agent configuration | Built-in local execution does not enforce this benchmark's exact configuration-discovery/catalog policy | Fresh HOME/Copilot/XDG/.NET/Aspire configuration per model trial; disables inherited personal/built-in skills, plugins, extensions, history and hosted GitHub MCP; checks actual skills/tool visibility against the variant |
| Application resources | The local backend stages workspaces, but does not define this application's service ownership or teardown | Fresh application state, unique Compose/container/volume identities, allocated ports, serial trials and targeted owned-resource cleanup, including failure/interruption |
| Security boundary | Vally also offers a separate native Docker backend | This benchmark currently uses the **local backend**, not Docker isolation; its controls are cooperative configuration/resource isolation, not an OS sandbox |
The executor checks the staged workspace location and hashes before inference.
Harness implementation and sibling variants are not staged into the agent's
application tree. Copilot authentication tokens are forwarded only through explicit in-memory
environment handling, not persisted in generated specs or environment files;
ambient harness context and TLS-bypass settings are excluded from the agent
environment.
**Host access remains possible.** The agent runs as your host user and can
access files outside its workspace, host tools, Docker and the network.
Processes that escape the owned working-directory tree may evade cleanup.
Ports are allocated but not reserved until services bind. The host Docker
daemon and its image cache are shared. These controls reduce accidental
cross-trial contamination; they do not contain malicious agents or fixtures.
Use a dedicated disposable machine for untrusted work, and treat logs and
trajectories as potentially sensitive.
### Tool installation versus per-trial state
The harness does **not** install .NET or Aspire between trials. It requires
the host's existing .NET 10 SDK and Aspire CLI 13.6.x:
- .NET uses the installed SDK with a fresh `DOTNET_CLI_HOME`.
- Each Aspire trial copies the existing CLI executable into its private
`home/bin`, without the installation sidecar, settings or credentials, and
uses a fresh `ASPIRE_HOME`. Raw retains host tools on PATH; Aspire is not
deliberately hidden from it.
- Application NuGet/npm restore, build and launch are the agent's task.
Fresh HOME/workspaces can trigger repeated dependency downloads; Docker
images remain cached in the shared daemon. This is not a guaranteed
cold-cache or warm-cache benchmark.
Fixture preparation and executable/config copying are setup, outside reported
agent time. Agent dependency restore/build/startup remain inside it.
The SDK session receives explicit application-workspace and runtime-ownership
context, identical in purpose across variants. Requested JSON files must be
written in that workspace, not HOME or the source staging copy; overriding
the supplied Compose project identity creates unrelated resources and fails
objective verification. These instructions reduce mistakes but do not enforce
an OS boundary. The runtime's selected model is checked before inference:
substitution of a named model fails explicitly instead of producing a
mislabeled benchmark. `auto` intentionally permits runtime model selection.
## Local setup
Use macOS or Linux with Bash, curl, `lsof`, Git, Node **24+**, a stable **.NET 10
SDK**, Docker with a running daemon and Compose v2, and **Aspire CLI 13.6.x**.
The initial harness supports local Unix Docker sockets, not Windows or remote
TLS Docker contexts. Install prerequisites through their official installers;
the harness does not install global tools or change global configuration.
```bash
git clone https://github.com/sebastienros/aspire-bench.git ~/github/aspire-bench
cd ~/github/aspire-bench
npm ci --ignore-scripts
npm run validate
npm run bench -- plan
npm run bench -- preflight
npm run bench -- dry-run
npm run bench -- smoke
```
`validate` type-checks, runs unit tests, and validates the registry and specs
against the pinned Vally API. CI runs only this offline validation: no auth,
Docker, or paid inference.
`dry-run` creates fresh workspaces and starts actual Copilot SDK sessions
**without sending a prompt**. It initializes the tool catalog, queries skills
and MCP servers, verifies their exact identities, and saves `visibility.json`.
Each variant must expose exactly its declared skills/MCP servers; the default
raw control exposes neither, and full Aspire exposes seven snapshotted skills
and live Aspire MCP tools. Unexpected inherited configuration
or missing tools fails closed. Requires Aspire, but not Docker or an inference
token.
`smoke` launches both real stacks serially **without an agent/model call**, runs
the common objective verifier, and cleans up even on failures/interruption.
It restores the injected startup configuration from the pristine snapshot in
its owned reference workspace before launch. This is a known-good repair smoke,
not a measurement of diagnosis; evaluation setup never performs that repair.
Startup may download public NuGet/npm packages and images. Smoke establishes
infrastructure readiness, not agent quality.
## Direct Vally evaluation
### Native local-snapshot experiment
[`experiments/bingo.experiment.yaml`](experiments/bingo.experiment.yaml) is the
comparison source of truth: native Vally `repo-comparison`, shared evals,
baseline `raw`, serial `execution.workers: 1`, and four varying axes:
`/environment/files`, `/environment/skills`, `/environment/mcpServers`,
`/environment/commands`.
Local directories are copied through Vally's `environment.files` contract.
There are no remote clones, new app repositories, or sibling/harness files in
an agent workspace. Paths in variant overrides resolve relative to the
experiment file. Vally resolves, hashes and rejects undeclared configuration
drift before any agent starts.
| Variant | Application snapshot | Aspire skills | Aspire MCP |
|---|---|---|---|
| `raw` | Raw, lifecycle-only README; no app setup instructions | No | No |
| `raw-documented` | Same raw app, manual configure/run/stop README | No | No |
| `raw-scripted` | Same raw app, lifecycle management scripts | No | No |
| `aspire-none` | Aspire | No | No |
| `aspire-mcp` | Aspire | No | Yes |
| `aspire-skills` | Aspire | Yes | No |
| `aspire` | Aspire | Yes | Yes |
All four Aspire cells use the **same AppHost/application snapshot**. This
supports AppHost-only comparison and skills/MCP ablation without assuming the
raw app supports Aspire MCP. "Skills/MCP" here means Aspire-specific additions,
not ordinary shell/file tools. Enabling skills also exposes the runtime's skill
loader. Configuration discovery and global skills/MCP remain disabled in all
cells.
Each of these seven variants also has a `-bugs` counterpart:
`raw-bugs`, `raw-documented-bugs`, `raw-scripted-bugs`, `aspire-none-bugs`,
`aspire-mcp-bugs`, `aspire-skills-bugs`, and `aspire-bugs`. Each counterpart has
identical files, guidance, skills and MCP, plus its app's Redis startup patch.
There are fourteen named variants, not a cross-product with invalid cells.
**Default selection remains the healthy `raw,aspire`: two trials per pair, not fourteen.**
`raw` has lifecycle-only guidance, not application setup instructions; the manual README is `raw-documented`,
and the preserved management scripts and their README are `raw-scripted`.
`--variants` selects explicit names or `all`; `--pairs` repeats that selected
set. Each repetition reverses variant order to reduce order bias.
```bash
# No inference: inspect resolved native plan and all fourteen effective catalogs.
npm run bench -- plan
npm run bench -- dry-run --variants all
# Paid commands: only run intentionally with a subscription token.
bash scripts/run.sh --model gpt-6-luna --variants aspire-none,aspire-mcp --pairs 1 --allow-paid
bash scripts/run.sh --model gpt-6-luna --variants all --pairs 1 --allow-paid
# Redis failure investigation and repair pair (paid, only when authorized).
bash scripts/run.sh --model gpt-6-luna --variants raw-bugs,aspire-bugs --pairs 1 --allow-paid
# No inference: verify the known-good repair and real workflow.
npm run bench -- smoke --variants raw-bugs,aspire-bugs
# Explicit raw guidance comparison (paid, only when authorized).
bash scripts/run.sh --model gpt-6-luna --variants raw,raw-documented,raw-scripted --pairs 1 --allow-paid
```
Every selection uses the same shared prompt, model, limits and objective grader.
Native arrays replace inherited arrays, maps deep-merge, and `null` clears
inherited MCP maps; offline tests verify these contracts and all seven native
healthy and bug staging/execution cells without inference.
### Post-start health questions
[`scenarios/health-checks.yaml`](scenarios/health-checks.yaml) is the sole active
scenario and the default. It uses Vally's native
[`turns`](https://microsoft.github.io/vally/reference/eval-spec/) configuration:
startup/repair prompt, followed sequentially by:
- "Are all services running and healthy?"
- "Is the database ready and healthy?"
- "Is redis ready and healthy?"
Each question also asks the agent to save its assessment in a separate file:
| Question | Result file | Required JSON |
|---|---|---|
| All services | `benchmark-services-health.json` | `{"running":true,"healthy":true,"evidence":"observed check and result"}` |
| Database | `benchmark-database-health.json` | `{"ready":true,"healthy":true,"evidence":"observed check and result"}` |
| Redis | `benchmark-redis-health.json` | `{"ready":true,"healthy":true,"evidence":"observed check and result"}` |
The booleans must reflect the observed state, and evidence must describe the
checks actually performed. The separate required native
[`custom-metrics`](https://microsoft.github.io/vally/reference/graders/custom-metrics/) graders
`services-health`, `database-health` and `redis-health` evaluate the files
produced by conversation turns 1, 2 and 3. Each parses its own JSON file and
asserts positive status booleans with typed `equals: true` checks, plus a
non-whitespace evidence string with `matches: '\S'`. A successful startup/repair is expected
to leave each target healthy; a negative assessment is retained but fails that
target's check. Missing, malformed or incorrectly typed metrics fail explicitly.
Each assertion has its own native result; partial credit cannot pass a required
grader. Additional keys are allowed, and native metrics also accept a `values`
envelope. This replaces the custom exact-key, 64 KiB and symlink checks with
Vally's native metrics semantics; configured paths must remain relative with
no `..` traversal. The harness remains cooperative isolation, not an OS sandbox.
Reports and the endpoint JSON are retained via Vally's native `artifacts` capture.
The health scenario also uses native
[`per-turn diff graders`](https://microsoft.github.io/vally/reference/graders/diff-contains/#per-turn-diff)
with zero-based `turn` selectors:
| Turn | Required output | Earlier outputs this turn must preserve |
|---|---|---|
| 0: startup | `benchmark-endpoints.json` | None yet |
| 1: all services | `benchmark-services-health.json` | Endpoint JSON |
| 2: database | `benchmark-database-health.json` | Endpoint and services JSON |
| 3: Redis | `benchmark-redis-health.json` | Endpoint, services and database JSON |
`diff-contains` requires each output to be written in its assigned turn;
`diff-not-contains` prevents subsequent questions from revising earlier answers.
Thus producing a missing report late, or correcting an earlier unhealthy answer
in a later turn, cannot pass even if the final JSON files and application are healthy.
Vally captures each turn's actual workspace diff through native turn-completion
callbacks; missing turn evidence fails rather than falling back to a full-run diff.
Vally 0.18 explicitly rejects `turn` on workspace-reading `custom-metrics` and
`program` graders. Those value checks run after the conversation on the preserved
files; they do not independently evaluate every conversational turn.
The separate live-application/cleanup check remains a whole-run check.
The `custom-metrics` graders also support artifact-directory grading after the ephemeral workspace is
gone; a malformed artifact fails rather than falling back to a workspace copy.
All four prompts use **one session and one running application per trial**.
Vally waits for each response before sending the next question; grading and
application cleanup occur after the conversation. If startup execution fails
or times out, later questions are not sent. Bug variants initially receive the
same Redis fault as before; health questions follow the agent's startup/repair
attempt, not a separately injected post-start fault.
The native experiment resolves **one eval × fourteen variants**. Startup is
already turn 0, so the redundant `launch-and-verify` scenario has been retired.
```bash
# No inference: inspect all fourteen health-check configurations.
npm run bench -- plan --scenario health-checks --variants all
# Paid: raw vs Aspire, with and without the startup bug.
bash scripts/run.sh --scenario health-checks --model gpt-6-luna \
--variants raw,aspire,raw-bugs,aspire-bugs --pairs 1 --allow-paid
# Paid: include every README/skills/MCP ablation and bug counterpart.
bash scripts/run.sh --scenario health-checks --model gpt-6-luna \
--variants all --pairs 1 --allow-paid
# Paid: run the full scenario/variant set with two models (28 trials).
npm run bench -- eval --model gpt-6-luna,gpt-5.5 \
--variants all --pairs 1 --allow-paid
```
Comma-separated models are passed directly to Vally's native `--model` matrix,
not expanded into separate harness runs. All models share one `.runs` folder
and dashboard export; each variant invocation schedules the models serially.
Every model receives its own HOME/config, ports, container/volume identity and
fresh application workspace. Baseline deltas are computed within a model only.
This is **one four-turn trial with three separately reported answer grades**,
not three independent application launches.
Recorded trajectory events retain their conversation-turn indexes; reported
time, tokens and tool calls cover startup plus all health questions, within
the shared fifteen-minute budget. Each file grader checks the structured
assessment, not the semantic correctness of its evidence prose. The required
`objective-success` grader independently checks the live application workflow
and cleanup; affirmative JSON cannot make an unhealthy application pass.
No LLM judge is added. Answers remain inspectable in Vally's trajectory and
captured JSON artifacts.
Native per-turn Git snapshots add host bookkeeping during execution; whole-run
elapsed time is not a per-question timing measurement. Re-grading per-turn diffs
requires their original recorded evidence and `--run-dir`; the compatibility
export preserves grader outcomes but does not copy auxiliary diff sidecars.
Vally also supports separate stimuli and turn-scoped graders, but separate
stimuli are separate trials, not continuations of an already-started session.
`apps/bingo/raw/` is the **one shared raw application snapshot**, with
build/dependency configuration, Compose for PostgreSQL/Redis only and a license.
It contains no README or lifecycle scripts. The native manifest composes each
agent workspace using `environment.files`; no duplicate app folders or runtime
edits are needed:
```yaml
# raw-documented; raw uses readmes/raw.md instead.
files:
- {src: ../apps/bingo/raw, dest: .}
- {src: ../apps/bingo/readmes/raw-documented.md, dest: README.md}
# raw-scripted uses readmes/raw-scripted.md and also copies:
# - {src: ../apps/bingo/scripts, dest: scripts}
```
Only the selected README is visible, always as `README.md`, and only
`raw-scripted` receives `scripts/`. Both `raw` and `raw-bugs` use the same
`apps/bingo/readmes/raw.md` overlay. It describes the application and instructs
detached long-lived services with retained logs/process identifiers, but provides
no application-specific setup commands or dependency instructions. The documented README
explains configuration, build/install, dependency readiness, migrations/seeding,
independent backend/frontend launches, endpoint submission and targeted stopping.
No raw variant receives an AppHost, skills, MCP, other READMEs, sibling fixtures
or host harness files. File composition is declared **only in the experiment
manifest**; the registry identifies the shared app/runtime kind, not a second
file-copy recipe.
Only `-bugs` cells receive the same Redis command-line bug **after copying**:
`--maxmemroy 64mb` (a misspelled `maxmemory` option). The raw patch modifies
`compose.yaml`; the Aspire patch adds the same arguments to Redis in `apphost.cs`.
Redis exits with a fatal configuration error. Shared source snapshots remain
healthy, and neither patch files nor an answer are staged for the agent.
The shared prompt asks only to start the application, verify all services,
investigate/fix startup problems, and leave it running. It supplies no technology,
file-name, README, diagnostic-method or fault hints; agents must discover the
application's orchestration themselves. The only reporting contract is the
admin/frontend URL JSON needed by the independent verifier. Healthy cells receive no patches,
so existing names/default selection remain intact. Compare healthy and fault
runs separately and use recorded commits and patch hashes for historical results.
All variants share the same task and endpoint/verifier contract. The shared
prompt does not require a documented entrypoint, so it also applies to lifecycle-only
`raw`. Setup and smoke infer manual versus scripted lifecycle from the selected
file overlays rather than maintaining a separate per-variant copy recipe.
`npm run bench -- smoke --variants raw,raw-documented,raw-scripted` verifies
all three owned real stacks without inference. Smoke uses host-only manual
reference commands for the two non-scripted variants, including documented stop;
it does not inject those commands or setup documentation into `raw`.
Evaluation setup never starts the app.
**No paid agent evaluation occurs during setup, validation, dry-run or smoke.**
A real evaluation requires an explicit consent flag and model:
```bash
# Token authorized for your Copilot subscription; never write it to a file.
export COPILOT_GITHUB_TOKEN="$(gh auth token)"
bash scripts/run.sh --model gpt-5.5 --pairs 3 --timeout 15m --allow-paid
unset COPILOT_GITHUB_TOKEN
```
The runner prints timestamped variant start/completion messages, repetition
numbers and a finished/total counter, also saved in `<run>/progress.log`.
Completion includes grading and cleanup; `failed` means the trial's lifecycle
exited unsuccessfully, not necessarily an agent execution error. Trials not
completed before interruption remain unfinished. Native Vally progress and
details remain in `<run>/<repetition>-<variant>/vally.log`; use `tail -f` on
that file for live detail (logs may contain sensitive content). Dashboard
exports are still created during final reporting, not continuously refreshed.
For multiple models, the visible counter counts variant batches: a batch
finishes after its Vally invocation and cleanup, and passes only if all model trials pass.
Alternatively export `GH_TOKEN` or `GITHUB_TOKEN`. The SDK does not inherit
your logged-in Copilot user, personal settings, or OAuth files. Token/account
policy must permit the selected model. Enterprise managed restrictions may
still apply; visibility validation rejects changes to the declared treatment.
Both variants receive the **same prompt, model, timeout, objective grader and
threshold** for each selected model. Runs are serial: one worker, one variant
per Vally invocation (one trial per selected model), and
**no automatic retries**. Pair order alternates raw-first/treatment-first.
Each trial has a new copied workspace, isolated HOME/Copilot config, fresh data
volume, unique containers/Compose project and allocated ports. Raw must discover
the launch steps; `raw-documented` has the manual README; `raw-scripted` has its
Bash launcher. Aspire uses exact-target
`aspire start --non-interactive --isolated` and `aspire wait`. Dependency restoration, builds and application
startup are still the agent's task, not pre-completed work.
Results default to ignored `.runs/<timestamp>/`; `--output DIR` requires a new
directory. Recreate the deterministic, **free** paired comparison with:
```bash
bash scripts/report.sh .runs/<timestamp>
```
| Artifact | Meaning |
|---|---|
| `metadata.json` | Source/harness commits, tool versions, OS/architecture, model, timeout, selected variants, recorded fixture/lifecycle definitions and baseline |
| `experiment.yaml`, `experiment-plan.json`, `<pair>-<variant>/plan.json` | Native experiment, resolved effective specs, hashes and declared varying axes |
| `paired.json`, `comparison.md` | Derived after cleanup from native JSONL and lifecycle status; unavailable failure metrics are N/A, not zero |
| `<pair>-<variant>/visibility.json` | Actual loaded skills and agent-visible tools checked before inference |
| `<pair>-<variant>/agent.json`, `proof.json` | Host executor identity/setup duration and program grader checks/grading duration |
| `<pair>-<variant>/vally.log`, `cleanup.log`, `exit-code`, `cleanup-exit-code` | CLI diagnostics and lifecycle completion; failed/incomplete cleanup cannot count as success |
| `<pair>-<variant>/session-logs/` | Exported SDK session history, including interrupted runs when available |
| Vally timestamped subdirectories | Native JSONL outcomes, Markdown report, SDK session logs and OTel trajectories |
| `<pair>-<variant>/workspace.json` | Retained disposable runtime identity, staged baseline hashes and optional patch paths/SHA-256 fingerprints |
| `vally-export/<snapshot>/` | Derived native experiment-format JSONL, real variant names, matched repetition indexes and lifecycle-aware verdicts for Vally's dashboard |
Multi-model trials retain per-model `workspace.json`, `agent.json`, `proof.json`,
session logs, native result records and lifecycle markers in
`<pair>-<variant>/model-<index>/`, following the recorded model-list order.
The parent keeps the original multi-model native results and combined log;
one failed model cannot overwrite another model's outcome or evidence.
Unstarted/interrupted model cells export as incomplete or execution errors,
never successes.
### Native Vally results dashboard
Use Vally's existing local dashboard, rather than a separate web app.
New `scripts/run.sh` runs automatically create a compatible export during final
reporting. Export an existing **single harness run folder** without changing its
original results or regenerating reports:
```bash
npm run bench -- export-vally .runs/<run>
# Prints .runs/<run>/vally-export/<snapshot>.
npx --no-install vally serve .runs/<run>/vally-export/<snapshot>
# Open http://127.0.0.1:3200.
```
A comma-separated multi-model evaluation needs only this single export.
Both model identities are retained for native filtering and comparisons;
there is no need to ingest separate model runs into SQLite.
The export has one native `results.jsonl` per named variant, so Vally's one-level
directory ingestion loads the **whole run**. The dashboard provides run/outcome
filters, metric matrices, grader/tool statistics and trajectory drilldowns.
It binds to `127.0.0.1` with CORS disabled by default. Keep those defaults.
There is no browser disk upload or cloud service.
To load multiple runs into the same dashboard, ingest each exported snapshot
into a local SQLite database, then serve the database:
```bash
npx --no-install vally ingest .runs/<run-a>/vally-export/<snapshot-a> --store .runs/history.db
npx --no-install vally ingest .runs/<run-b>/vally-export/<snapshot-b> --store .runs/history.db
npx --no-install vally serve --store .runs/history.db
```
No model credits are used by export, ingestion or serving (do not add
`--compare` to an evaluation: Vally's comparison judge is a different, paid
operation). Source files remain untouched. Exports are immutable content-based
snapshots; exporting unchanged inputs again returns the same directory. If an
interrupted/in-progress run later acquires more artifacts, exporting again
creates a **new snapshot**. Ingest only the desired snapshot per source run,
not every revision, or the history will contain multiple views of the same trials.
Vally does not live-refresh or re-ingest an existing database run identity.
The adapter preserves native trajectories/usage and replaces the per-invocation
`main` label with the recorded harness variant. `trialIndex` is zero-based from
the recorded repetition. Healthy and `-bugs` cohorts have separate experiment
identities and the corresponding `raw` / `raw-bugs` baseline. Identities and
visible eval/experiment labels include cohort/model/harness commit; no histories,
models or task versions are silently pooled or relabeled. Vally's comparison UI
still allows you to manually select unrelated runs: check provenance before
drawing conclusions. It does not implement the harness's matched-success-only
delta policy; use `comparison.md` for those deltas.
For current scripted runs the exported verdict requires the native grade,
objective proof, and successful trial **and cleanup** exit markers. The additional
`harness-lifecycle` grader shows execution errors, objective failures, cleanup
failures or incomplete evidence. Legacy exports preserve the original
`paired.json` verdict, including known historical harness defects, rather than
retroactively declaring success. `export.json` and the JSONL's `harness` field
retain definitions, provenance, missing-metric flags and setup/grading timing.
Malformed or conflicting artifacts fail export explicitly.
**Native UI limitation:** Vally 0.18 normalizes missing trajectory usage to zero
in some charts. The exporter does not invent metrics; its lifecycle grader
evidence flags unavailable metrics explicitly. An execution-error duration may
be invocation time, not measured agent time. Do not interpret those zero bars as
zero-cost failures; use the harness's N/A-aware report for cost comparisons.
Relative native diff sidecars are not copied; original trial folders remain the
source for auxiliary artifacts.
**Privacy:** exports retain original local trajectory content and can contain
credentials or private output. They are not redacted. Keep the dashboard local,
do not commit exports/databases, and review/redact artifacts before sharing.
Vally 0.18's native `vally compare` invokes a paid prompt judge; its experiment
runner parses `grader_plugins`, `executor_plugins` and `eval_plugin` but does
**not load them**. Running `vally experiment run` directly would lose required
isolation hooks. Small lifecycle scripts therefore use the native
`resolveExperiment` API (including merge/drift/hash validation), then invoke
**`vally eval` directly**, with `--executor-plugin`, one worker and no retries.
Both scenarios share two required checks using Vally's **built-in `program` grader**:
`endpoint-contract` runs `scripts/verify.sh endpoint-contract` to validate the
generated JSON, and `objective-success` runs `scripts/verify.sh` to verify the
live application and clean up. There is no custom grader plugin or in-memory
proof map. The health scenario additionally declares the built-in per-turn
diff checks and three `custom-metrics` report graders described above.
Plugin fields are deliberately not placed in the manifest.
`endpoint-contract` reports separately in native results. It requires
`benchmark-endpoints.json` to exist, parse as JSON, and contain exactly `admin`
and `frontend`: distinct loopback HTTP origins with explicit ports, no
credentials, paths, query strings or fragments. It reuses the objective
verifier's endpoint validator; it neither starts/stops services nor checks
their health. Valid JSON alone cannot pass an evaluation: both graders remain
required, and `objective-success` still validates the endpoint contract itself.
`scripts/run.sh` selects/repeats named variants, alternates order and calls
`scripts/setup.sh` followed by `scripts/trial.sh`. Setup copies/configures but
does **not** launch the application. The trial script calls the pinned local CLI:
```bash
node node_modules/@microsoft/vally-cli/dist/index.js eval \
-e "$TRIAL/eval.yaml" --work-dir "$RUNTIME/app" \
--workspace "$RUNTIME/workspaces" --output-dir "$TRIAL" \
--workers 1 --max-retries 0 --require-pass \
--executor-plugin "$PWD/dist/plugin.js" --shutdown-timeout 3m
```
For multiple models the script adds native `--model MODEL1,MODEL2` and uses
Vally-managed temporary workspaces under the owned runtime's `TMPDIR`.
Vally 0.18's explicit `--workspace` path otherwise collides across models.
Temporary workspaces are released after grading; captured JSON artifacts,
turn diffs and trajectories remain in the original native results directory.
The lifecycle deadline scales with the number of selected models.
Native file/patch setup runs before the executor receives a model identity.
Setup therefore uses a stable host-owned context (`ASPIRE_BENCH_SETUP_OWNERSHIP`)
for the common variant's fingerprinted patches and confined native workspace
root, independently of the execution/grading context selected per model.
Setup never starts services or supplies another model's HOME/ports. This avoids
stale ownership from a preceding model contaminating the next setup command.
Pre-fix multi-model bug trials that failed with
`Patch setup must run in the owned application workspace` are harness setup
errors, not evidence of model repair ability; their recorded results are not
rewritten. Model-authored source/lockfile changes still fail integrity grading.
Use `scripts/trial.sh`, rather than pasting this command into an ambient shell:
it passes the isolated runtime environment, keeps credentials only in memory,
and traps failure, interruption and a bounded lifecycle deadline.
The program grader independently verifies the still-running application and
cleans owned resources; the shell finalizer calls `scripts/cleanup.sh` again
even if execution fails or grading never runs. `scripts/report.sh` reads native
JSONL after cleanup; reporting does not orchestrate inference. The existing
`npm run bench -- eval ...` command is a compatibility alias for `scripts/run.sh`.
Vally owns local file/skill staging, inference, normalized metrics, trajectories
and native reports. The scripts only supply per-run owned HOME/services/auth,
retain evidence and produce a free comparison. The minimal executor controls
SDK auth/config discovery and asserts staged inputs/effective catalogs before
inference; arbitrary setup scripts cannot enforce those session-level controls.
Vally's default executor enables configuration discovery and does not expose
our exact skills/extensions/history restrictions or catalog assertions.
The scripts do not independently reconstruct variant specs from registry
fixtures. Runtime-generated IDs/ports/homes are unique per trial rather than
experimental factors; secrets remain outside manifest artifacts.
Reports separate healthy and `-bugs` trials: healthy variants compare to `raw`,
and bug variants compare to `raw-bugs` only when it was selected. They never
compute deltas between different fault conditions. The native manifest baseline
remains `raw`; `raw-bugs` is the matching reporting control for the fault cohort.
Compare objective success first, then costs among successful trials. Cheap
failures are not improvements. Tokens are SDK/Vally usage metrics, not billing
estimates; inspect raw usage events for missing telemetry. Report sample size,
failures, model, timeout and environment. One pair is an infrastructure check,
not a statistically meaningful finding. The first local Luna pair was inconclusive:
raw timed out and an already-fixed cleanup observer defect invalidated the
Aspire verdict despite passing application checks. Those private local
artifacts are retained unchanged, not published as a benchmark claim.
**Historical runs named `raw` used scripted guidance before `319104f`**,
including the original Luna runs. At `319104f`, `raw` used the manual README;
From `caac578` through `35dbfde`, `raw` had no lifecycle guidance. The current
`raw` and `raw-bugs` have lifecycle-only detached-process guidance. Do not relabel
or reinterpret older runs as today's raw.
Reports use recorded metadata/provenance, never the current registry to infer
past variant meanings. New runs record lifecycle and effective file definitions explicitly; reports
of older runs flag the naming boundary rather than silently resolving old `raw`
against today's manual fixture.
## Patch-based variants
Vally 0.18 supports **`environment.commands`**, executed after files/skills are
staged and **before** workspace baselines and agent execution. There is no
separate middleware API needed here; an executor wrapper would apply the patch
too late for native diff attribution. The harness now supports a constrained
setup helper, `dist/patch.js`, for applying Git-format text diffs to copied apps.
The current experiment's `-bugs` variants use `apps/bingo/patches/raw-redis-startup.patch` and
`aspire-redis-startup.patch` for equivalent failures in the two orchestration
formats, without duplicating either application.
To add a future patched variant, create a `.patch` or `.diff` file in the
repository, register the variant against the existing app/runtime kind, and
declare the patch command axis and setup command in the native manifest:
```yaml
vary:
- /environment/files
- /environment/skills
- /environment/mcpServers
- /environment/commands
variants:
raw-with-patch:
environment:
files:
- {src: ../apps/bingo/raw, dest: .}
- {src: ../apps/bingo/readmes/raw.md, dest: README.md}
skills: []
mcpServers: null
commands:
- 'node "$ASPIRE_BENCH_ROOT/dist/patch.js" "$ASPIRE_BENCH_ROOT/apps/bingo/patches/change.patch"'
```
This is an example fragment, not an additional enabled cell. The root variable
is host-side lifecycle context supplied by `scripts/trial.sh`; patch paths are
**repository-root-relative**, unlike `environment.files` sources. Only this
helper command form is accepted: arbitrary shell/setup commands remain blocked.
For several patches, list helper invocations in application order.
Preparation applies the same sequence to its owned reference copy and records
each patch's SHA-256. Native Vally setup applies it to the actual trial workspace
before inference, rejecting changed inputs. The original app is never modified,
and patch files/helper scripts are not copied into the agent workspace.
The intentionally patched tree becomes the initial source baseline;
setup time and setup edits are not charged or attributed to the agent.
Each patch is checked with `git apply --check` before applying it. Invalid or
nonapplicable hunks, path escapes, symlinks/submodules, binary patches and
protected configuration/runtime paths fail before inference. Renames/copies
are not supported. Text-file
modification, addition and deletion are supported; earlier valid patches in a
sequence may remain in the disposable copy if a later patch fails, but no agent
starts and the shared source stays unchanged. No services are started by patch
setup.
Patch application is independent of the task/grader. The current scenario
permits edits only to patch-modified startup configuration (`compose.yaml` or
`apphost.cs`), and requires that configuration to change from the broken
baseline. Correcting the option or removing it can both pass; grading does not
require one exact diff. All other pre-existing source/guidance remains immutable,
and the complete real application workflow must pass. Future bugs in service
source need an explicitly repair-aware scenario rather than weakening this
startup-only allowance.
## Objective success
The agent leaves the stack running and writes loopback admin/frontend origins
to `benchmark-endpoints.json`. Its self-report is not evidence. Before cleanup,
the host verifier checks:
1. Injected startup configuration is repaired; other fixture source/guidance is
unchanged; endpoints belong to newly created
run-owned processes; dependency containers belong to this run.
2. PostgreSQL accepts connections; actual Identity/BingoSquare tables contain
migration and seed data; Redis answers authenticated PING when required.
3. Real admin login HTML and player frontend are accessible. The frontend API
proxy reaches the backend and its SignalR proxy negotiates transports.
4. A fresh square can be imported into PostgreSQL, called, observed in
Redis-backed producer status and cleared through the application's developer
API. Both variants receive the identical workflow.
Timeouts, unrelated modified inputs, unrepaired startup configuration, mocks, missing endpoints/host evidence, failed
workflow assertions and cleanup failures cannot count as success. Grading is
not based on agent-answer greps, health endpoints alone or a file's presence.
## Treatment, timing and limitations
All seven upstream Aspire `SKILL.md` files and **all their references** are
copied project-locally by native `environment.skills`, one directory per skill
at the workspace root (Vally 0.18's supported layout). Upstream skill-evaluation
fixtures are excluded. This is a complete licensed guidance snapshot, not
hand-written substitutes. `aspire agent init` is the supported regeneration
path for future snapshots; never copy personal skill/config directories.
Treatment has `.github/mcp.json`, but the SDK explicitly receives
`aspire agent mcp`: Vally does not discover workspace MCP JSON.
The minimal executor delegates inference/metrics to Vally's Copilot SDK executor,
controls discovery, disables personal/built-in skills, plugins, extensions,
cross-session history and hosted GitHub MCP, and checks the effective catalog
before inference. Raw has no AppHost, active ServiceDefaults, Aspire skills or
checkpoint documentation. Original Bingo content/version fields may mention
Aspire; those are application data, not orchestration guidance.
Fixture copying, executable/config isolation and visibility checks are setup;
verification/cleanup are post-task work. These are excluded from reported agent
elapsed time. Agent restore/build/launch remain included. Native SDK/session
startup overhead remains in Vally time. Setup and grading durations are recorded
separately.
This is **configuration/resource isolation, not an OS security sandbox**.
The cooperative local agent has shell access; Aspire is not hidden from raw's
PATH. Objective checks reject unrelated source/guidance changes and verify
owned services and real application behavior, but the intentionally minimal
prompt is not a safety boundary. A malicious agent can
reach host files, Docker or the network. Use trusted fixtures/skills, preferably
on a dedicated disposable machine. Tokens are passed only through the child
environment, not saved in specs. Session logs may contain sensitive output:
review/redact all runtime artifacts before publishing them.
NuGet/npm caches, image downloads, file-based .NET SDK caches, CPU contention
and network variability affect startup. Versions/service tags are pinned, but
image tags are not immutable digests. The default raw/full-Aspire pair bundles
AppHost, health/telemetry, skills and
MCP, so it cannot attribute differences to one component. Select the named
Aspire ablation cells to measure skills and MCP separately; AppHost versus raw
still bundles orchestration, health and telemetry.
Script-installed Aspire may select its install sidecar before `ASPIRE_HOME`.
The harness copies only the installed executable, without that sidecar, into
the disposable treatment home; no settings or credentials are copied.
HTTP profiles avoid certificate trust prompts and automatic certificate
generation is disabled. Short macOS runtime paths accommodate Unix sockets.
## Cleanup and recovery
Cleanup first stops the **exact** Aspire AppHost, never `--all` or `--force`.
It then terminates only newly created processes whose kernel-reported cwd is
inside the owned runtime root, and removes only exact run-owned containers,
volumes and Compose networks. No global prune or name-based process killing.
SIGINT/SIGTERM trigger cleanup; SIGKILL or host crashes cannot.
Runtime directories/logs are retained for diagnosis. After a hard crash, use
the exact runtime root recorded in `workspace.json`:
```bash
bash scripts/cleanup.sh /tmp/aspirebench-<exact-run-directory>
```
This removes that run's services/disposable data, not retained files. The
host-generated ownership manifest is trusted local state; never edit it or
point cleanup at an arbitrary directory.
## Add an application or scenario
Add licensed, self-contained snapshots under `apps/<name>/`, record
source commits/adaptations/licenses in `provenance.json`, and register variants
and scenarios, including an explicit `adapter` and native `experiment` path, in
`apps/registry.json`. Add a manifest under `experiments/` declaring local staging,
baseline, serial workers, and only intentional `vary` axes.
Reuse the same registered raw `fixture` for guidance variants; keep README
overlays in sibling `readmes/` and optional lifecycle scripts in sibling
`scripts/`. Declare each composition in native `environment.files`, always
copying the common source first, one README to `README.md`, and scripts only
when intended. Do not duplicate application sources just to change guidance.
Register its launch/verifier implementation in `harness/adapters.ts`; unknown
adapters fail closed. Add one common Vally spec under
`scenarios/`, not different objectives for control and treatment. Setup
discovers apps/scenarios from the registry and resolves variant specs from the
native manifest. Keep variant-selection defaults explicit so adding a cell
does not silently increase evaluation spend.
A new architecture needs its own smoke/verifier adapter and cleanup ownership
contract. Do not silently grade it using Bingo endpoints/tables. Add positive
and negative verifier/isolation/cleanup tests, then run `validate`, `dry-run`
and non-agent `smoke` before spending evaluation credits.