redhat-et/parsec-bench

Latest recorded source activity: October 26, 2026 at 12:43

SkillsMP catalog refreshed: November 5, 2026 at 19:57

skills collected: 5

GitHub stars: 2
GitHub forks: 2

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/redhat-et/parsec-bench.

Repository README

Repository README source: README.md

Read the source README

Expand full README

# PARSEC simulation benchmark 200 hand-authored tasks that measure PARSEC, a Red Hat operations agent, against five LLM-simulated backend services. Each task is a [Harbor](https://github.com/harbor-framework/harbor) task: an instruction, per-service seed data, and a contract that its own verifier scores. This directory holds everything needed to read the tasks, audit the measured baseline, and reproduce it. | document | read it for | |---|---| | [`ARCHITECTURE.md`](ARCHITECTURE.md) | how the simulations, the parsec patch, Harbor and the verifier fit together | | [`TASKS.md`](TASKS.md) | the task categories, their counts and baseline means, and a one-line catalogue of every task | | [`BASELINE.md`](BASELINE.md) | the measured baseline: setup, the headline mean, and every task's score | | this file | the layout, and step-by-step instructions to stand up the environment and replicate the baseline | ## Layout ``` README.md ARCHITECTURE.md TASKS.md BASELINE.md tasks/<task-id>/ the 200 Harbor tasks instruction.md the prompt given to the agent, verbatim task.toml declared services, image, MCP server URLs, timeouts seeds/<service>.json the world state, one file per declared service tests/expected.json the contract (schema "bench/v4") tests/verify.py the scorer; identical in every task tests/test.sh Harbor's verifier entrypoint golden.json an authored reference solution that scores 1.0 provenance.md what the task measures, where every literal comes from bundles/parsec-<service>/ the five simulation bundles api.json schema.json db.json SKILL.md baseline/<task-id>/ the scored trajectory of every measured task agent/agent.jsonl the transcript the verifier read verifier/reward.json the four scores verifier/reward-detail.json which fact, call or trap decided each score baseline/results.json every task's metadata and score, machine-readable patch/ parsec-simulation-redirect.patch points parsec's tools at the simulations scripts/ parsec_harbor_agent.py the Harbor agent: drives patched parsec, writes the transcript install_seeds.py installs one task's seed data into the running simulations ``` Task ids carry a `bench-v4-` or `bench-v5-` prefix. The prefix records which of two authoring rounds produced the task and has no other meaning: both rounds use the same contract format, verifier, bundles and agent, and the benchmark is the 200 tasks together. The rest of the id names the primary service, a number, and a slug. The two scripts in `scripts/` use only the Python standard library, plus `harbor` and `httpx` for the agent. Both are short and commented; read them before running them. ## What you need - **Harbor 0.22** (`pip install harbor==0.22.0`, or from its repository). - **The simulation harness at commit `4f6ecca` (v0.2.0)**, from <https://github.com/skillberry-ai/simulation-harness>. The version matters: the bundles were generated by v0.2.0, and two of its behaviours are load-bearing. It honours a storage-scope filter, where v0.1.2 ignored one it had no field for. And `fetch_github_file` returns an in-band error for a path no seed holds, where v0.1.2 invented a plausible file. On an older harness, several tasks read data that is not there, or reason over content nobody seeded, and score confidently wrong. - **parsec at commit `28e2c40`**, from <https://github.com/rhpds/parsec>. Every task's provenance cites line numbers in that tree. - `podman` or `docker` with compose, Python 3.11 or newer, and `uv` (for parsec). - **An OpenAI-compatible endpoint for the simulations.** The harness asks a model to write every simulated tool response. The baseline used `azure/gpt-5.4`. - **An Anthropic-compatible endpoint for parsec.** The baseline used `claude-sonnet-4-6` through a LiteLLM gateway, which can also serve the first role. - **Time.** A model writes every simulated tool response. Calls took 4–19 s on a quiet gateway and 30–60 s under load, so a five-call task runs for minutes. The baseline's agent time alone summed to 8.1 hours over the 194 scored tasks. ## 1. Get the two upstream checkouts Work in one directory that holds this distribution, unpacked as `parsec-bench/`, and the two clones. ```bash git clone https://github.com/skillberry-ai/simulation-harness git -C simulation-harness checkout 4f6ecca # v0.2.0 git clone https://github.com/rhpds/parsec git -C parsec checkout 28e2c40 ``` Build the harness image once. Alternatively, pull `ghcr.io/skillberry-ai/simulation-harness:v0.2.0`. Do not use `:main`, which tracks unreleased code. ```bash docker build -t simulation-harness:bench ./simulation-harness ``` ## 2. Bring up the five simulations **Credentials.** Put the simulation endpoint in a `.env` beside your compose file, not inside the harness clone. The image never copies a `.env`, so a file inside the clone is read by nothing. ```bash cat > .env <<'EOF' LLM_API_KEY=... LLM_API_BASE=https://your-openai-compatible-gateway HARNESS_LLM_SIMULATION_MODEL=azure/gpt-5.4 EOF ``` `HARNESS_LLM_SIMULATION_MODEL` is sent verbatim as the `model` field, so use the name your endpoint expects. You do not need a skill-generation model, because the bundles ship pre-generated. A model that refuses `temperature: 0` fails outright, and so does one that caps completion tokens below 20000. **Topology.** A harness process hosts at most one simulation, so the five services need five instances. Each listens on 8086 inside its container; only the published port differs. `install_seeds.py`, the patch and the tasks all assume these ports. | service | port | tools | |---|---|---| | platform | 8086 | `query_aap2`, `query_splunk`, `query_ocpv_cluster`, `query_babylon_catalog` | | github | 8087 | `fetch_github_file`, `lookup_catalog_item`, `search_github_repo`, `search_agnosticv_prs` | | icinga | 8088 | `query_icinga` | | cost | 8089 | `query_aws_costs`, `query_gcp_costs`, `query_azure_costs`, `query_aws_capacity_manager`, `query_cost_monitor` | | cloud | 8090 | `query_azure_pools`, `query_gcp_projects`, `query_cloudtrail`, `query_aws_account`, `query_aws_account_db`, `query_marketplace_agreements` | ```yaml # compose.yaml x-harness: &harness image: simulation-harness:bench environment: LLM_API_KEY: ${LLM_API_KEY:?set LLM_API_KEY in .env} LLM_API_BASE: ${LLM_API_BASE:-} HARNESS_LLM_SIMULATION_MODEL: ${HARNESS_LLM_SIMULATION_MODEL:-azure/gpt-5.4} HARNESS_SKILLS_FOLDER: /app/skills-store HARNESS_AUTOSTART_ENABLED: "false" # each bundle is started explicitly HARNESS_SESSIONS_IDLE_TIMEOUT_SECONDS: "86400" # see below volumes: # Shared by all five. Each instance writes only its own parsec-<service>/. - ./skills:/app/skills-store restart: unless-stopped services: harness-platform: { <<: *harness, container_name: harness-platform, ports: ["8086:8086"] } harness-github: { <<: *harness, container_name: harness-github, ports: ["8087:8086"] } harness-icinga: { <<: *harness, container_name: harness-icinga, ports: ["8088:8086"] } harness-cost: { <<: *harness, container_name: harness-cost, ports: ["8089:8086"] } harness-cloud: { <<: *harness, container_name: harness-cloud, ports: ["8090:8086"] } ``` **The idle timeout.** The harness reclaims a session left idle for longer than its timeout, which defaults to 3600 s. A service can easily sit untouched for an hour while other tasks run. A reclaimed session answers `{"error": "Session expired: idle_timeout_exceeded ..."}`, which scores as the agent's tool failure rather than the infrastructure failure it is. The compose file above raises the timeout to 24 h. If you run the harness some other way, set it there. **Copy the bundles into the mount, start the instances, and activate one bundle on each.** Copy rather than symlink, because the harness writes seed data into this directory. ```bash mkdir -p skills && cp -r parsec-bench/bundles/parsec-* skills/ docker compose up -d for p in 8086:platform 8087:github 8088:icinga 8089:cost 8090:cloud; do curl -fsS -X POST "localhost:${p%%:*}/api/v1/simulation/start" \ -H 'content-type: application/json' -d "{\"name\": \"parsec-${p##*:}\"}" done ``` Activation returns `202`. Poll `GET /api/v1/simulation` on each port until `status` is `ready`. Starting a second simulation in one instance returns `409`, which is why there is one instance per service. Under podman every container reports `(unhealthy)` while serving traffic correctly, a healthcheck-translation quirk. Ignore that column and check over HTTP: `curl -s localhost:8086/api/v1/simulation/status`. ## 3. Patch and start parsec Patch a separate clone, and leave the step 1 checkout unmodified so the provenance line numbers stay valid: ```bash git clone --shared parsec parsec-live && cd parsec-live git checkout 28e2c40 git apply ../parsec-bench/patch/parsec-simulation-redirect.patch ``` The patch adds `src/parsecsim/` and changes two files. [`ARCHITECTURE.md`](ARCHITECTURE.md) explains what each change does. The one operational consequence: **you do not configure Icinga, Splunk or any cloud credentials.** With `PARSEC_SIM` set, the patch exposes every routed tool and answers every call from the simulations. Startup refuses to proceed if any routed tool would still be hidden: ``` RoutedToolsMissing: simulation is on but parsec would not expose every routed tool: gated off by an unconfigured backend: ['query_icinga'] ``` Install parsec, including four runtime dependencies its `pyproject.toml` omits: ```bash uv venv --python 3.12 uv pip install -e . uv pip install mlflow-skinny azure-cosmos google-cloud-resource-manager 'mcp>=1.8.0' ``` Configure the model in `config/config.local.yaml`: ```yaml anthropic: backend: "litellm" # or "api" with an Anthropic key model: "claude-sonnet-4-6" litellm_base_url: "https://your-gateway" litellm_api_key: "..." max_tokens: 4096 max_tool_rounds: 12 agent: runtime: "legacy" # the dispatch path the redirect hooks auth: allowed_groups: "" # disables the SSO gate for local runs allowed_users: "" ``` Start it: ```bash PARSEC_SIM=1 \ PARSEC_SIM_TRACE=/tmp/parsec-trace.jsonl \ PARSEC_SIM_TIMEOUT=240 \ MLFLOW_TRACKING_URI= \ uv run uvicorn src.app:app --host 127.0.0.1 --port 8000 ``` | variable | purpose | |---|---| | `PARSEC_SIM` | Turns the redirect on. Unset, nothing is redirected, so a real deployment can never be pointed at a simulation by accident. | | `PARSEC_SIM_TRACE` | **Required.** The redirect appends one JSON line per tool call here. The agent builds the trajectory from this file, so **parsec and Harbor must name the same path**. If it is unset, or the two paths differ, every task scores as if no tool was called. | | `PARSEC_SIM_TIMEOUT` | Seconds allowed per simulated call (default 300). Without a ceiling, one hung simulation stalls a run indefinitely; with one, the agent sees a tool error and moves on. | | `<SERVICE>_MCP_URL` | Optional per-service override of `http://localhost:<port>/mcp/sse`. | Startup warnings such as `AWS initialization failed` are expected: no cloud credentials are configured, and none are reached. Wait for `Application startup complete`. ## 4. Run one task **Seed first, every time, for every task.** Each task's seed replaces the previous one. A task run against the previous task's data scores plausibly and wrongly. ```bash python3 parsec-bench/scripts/install_seeds.py parsec-bench/tasks/bench-v4-platform-002-collection-not-found-rca ``` The installer writes each declared service's seed, then reads it back from both the harness's file view (`/api/v1/simulation/database`) and its in-memory store (`/api/v1/simulation/state`). It refuses unless both equal the seed exactly. It also installs an **empty store** (every collection in the bundle's `schema.json`, as `[]`) on every live service the task does not declare. parsec calls any of its tools regardless of the task, and without the reset an undeclared service would answer from the previous task's seed. It refuses a declared service that is unreachable, a seed for a service the task does not declare, and a service that is still starting. Then run the task with Harbor from `parsec-bench/`, with `scripts/` on `PYTHONPATH` so Harbor can import the agent: ```bash cd parsec-bench PYTHONPATH=scripts \ PARSEC_URL=http://127.0.0.1:8000 \ PARSEC_SIM_TRACE=/tmp/parsec-trace.jsonl \ PLATFORM_MCP_URL=http://localhost:8086/mcp/sse \ GITHUB_MCP_URL=http://localhost:8087/mcp/sse \ harbor run --path tasks/bench-v4-platform-002-collection-not-found-rca \ --agent parsec_harbor_agent:ParsecAgent \ --n-concurrent 1 ``` - **`PARSEC_URL` and `PARSEC_SIM_TRACE` are both required.** Whatever answers on `PARSEC_URL` is what gets scored, and `PARSEC_SIM_TRACE` must name the file parsec was started with. - **Export one `<SERVICE>_MCP_URL` per service in the task's `task.toml`.** They are templated there. - **Run one task at a time** (`--n-concurrent 1`). The simulations hold one task's seed, and the trace file holds one run's calls. The agent runs on the host, inside Harbor's process. The task container runs only the verifier: `tests/test.sh` calls `tests/verify.py`, which writes `reward.json` and `reward-detail.json` under the trial's `verifier/` directory in `jobs/`. ## 5. Read the result `reward.json` holds four numbers: | field | meaning | |---|---| | `completion` | A gate. 1 if the run finished with a non-empty answer, else 0, which zeroes the reward. | | `tool_calls` | The fraction of expected calls matched, arguments included. **Zero if any forbidden call was made.** | | `answer` | The fraction of answer items satisfied: required facts, counts, citations and forbidden facts together. | | `reward` | `completion × (w_tool × tool_calls + w_answer × answer)`, with the task's own weights. | Read `reward-detail.json` before drawing a conclusion. It names each missed fact, each forbidden item hit and each unmatched call. A `tool_calls` of 0 can mean that no expected call matched, or that a trap fired. Those are very different findings. ## 6. Replicate the baseline Repeat step 4 for every task: seed, run, collect. Do not seed once and run many. Then compare per task against [`BASELINE.md`](BASELINE.md), or against `baseline/results.json` for scripting. Every baseline row links to the transcript and `reward-detail.json` behind it, so you can diff your run against ours call by call. Three things to know before you compare: - **One reward is one sample.** Every tool response is written by a model, so a task's reward moves between runs with nothing changed. One task scored 0.420 four times and 0.520 once. Re-run a task before concluding anything from a single difference. - **Compare tasks, not only the mean.** The tasks were chosen to exercise all twenty modelled tools, not to mirror real PARSEC traffic, so the mean is not workload-weighted. - **If your model gateway caches whole responses, an identical repeated call is not a fresh sample.** Ours did. A re-run is independent only where the agent's calls differ from the first run's. ## Pitfalls we hit - **Parsec's routing limits what a task can measure.** Parsec hands each question to a sub-agent, and each sub-agent has a restricted tool set. A question routed to a sub-agent that lacks the expected tool cannot make that call at all. [`BASELINE.md`](BASELINE.md) labels the 24 baseline rows where this happened. A low `tool_calls` may be measuring routing, so check the trajectory. - **Parsec mounts its routes under `/api`.** The agent posts to `/api/query`. A `POST /query` returns 405, which matters only if you drive parsec yourself. - **Parsec's `mcp` may resolve to 1.x**, which spells the tool-error flag `isError` rather than `is_error`. The redirect reads both, and anything you write against `CallToolResult` should too. - **The skills directory ends up owned by the container user.** If a copy or cleanup fails with `Permission denied`, stop the containers and remove the directory as that user (`podman unshare rm -rf skills`). - **If you write your own seeds:** read the bundle's `SKILL.md`, which names the collection each operation reads and how it resolves an omitted argument. Give every row in a collection a distinct primary key, because the store keeps only the last row per key. If a tool scopes by a field (a cluster or a namespace), every seeded row must carry it, or the simulation returns nothing for a record that is plainly there. Replay the golden calls against the live simulation and check that nothing comes back empty: no offline check catches a missing scope field. ## What the benchmark claims, and what it does not - **It measures agent capability on three axes:** tool-call correctness including arguments, grounded synthesis without fabrication, and dependent multi-step chains. - **It does not establish simulator fidelity.** The simulations are grounded in parsec's own source, and every task's golden calls were replayed against the live simulations before it was measured. A task whose replay diverged in a way that changes what can be graded is UNMEASURED, and one that diverged without changing that carries a note. But an agent's arguments differ from the golden path's, so read a trajectory against the divergences listed in [`BASELINE.md`](BASELINE.md). - **No task contains production data.** Every literal is either attested in parsec's source at `28e2c40` or declared as invented in that task's `provenance.md`.

Loading the repository README…