invertase/reeval

Dernière activité source enregistrée: 4 novembre 2026 à 19:51

Catalogue SkillsMP mis à jour: 5 novembre 2026 à 19:57

skills collectés: 39

Étoiles GitHub: 13
Forks GitHub: 0

Installer avec un assistant IA

Copiez ce prompt dans l’assistant IA que vous utilisez.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/invertase/reeval.

README du dépôt

Fichier source du README : README.md

Lire le README source

Afficher le README complet

``` ██ ▄▄ ▄▄▄ ▄▄▄▄ ▄▄▄▄ ▄▄ ▄▄ ▄▄▄▄ ██ ███▀▀▀ ██▄▄██ ██▄▄██ ██ ██ ▄▄▄██ ██ ██ ██ ▄ ██ ▄ ████ ██ ██ ██ ██ ▀▀▀▀ ▀▀▀▀ ▀▀ ▀▀▀▀▀ ▀▀ Evaluation runner for SDKs, CLIs, MCP servers, and docs. # Get started using the `/setup` skill: npx skills add invertase/reeval --skill setup # Or install manually: npm i -D reeval && npx reeval init ``` > [!IMPORTANT] > **Work in Progress**: reeval is under active development. Expect breaking changes between minor version releases. # reeval reeval is an open-source evaluation runner for developer tools. It gives a real coding agent a task in a fresh sandbox, then checks the result with assertions on files, commands, and the transcript, and with an LLM judge where an assertion can't decide. Run the same task with and without a skill, doc, or MCP server, and reeval reports whether the change helped. Evals are TypeScript files in `evals/`, next to the tool they test. Each run writes a JSON record, and recorded runs commit to Git, so `reeval compare` can diff a pull request's run against the last recorded one and fail the job when an eval regresses. ## Features Cursor ⋅ Claude Code ⋅ Codex ⋅ Antigravity ⋅ OpenCode ⋅ Goose ⋅ pi ⋅ Grok Build - **Real agents**: each harness runs the agent itself, with its own tools, skills, and rules, as it works on a developer's machine ([Choose an agent](docs/configuration/agents/index.mdx)). - **Keys stay on the host**: each attempt starts in a fresh sandbox that holds only a placeholder key. reeval's egress gateway swaps in the real key on requests to that provider and nowhere else ([Pass secrets](docs/configuration/environments/secrets.mdx)). - **Compare a change**: one eval runs a baseline and its variants with the same prompt and checks, then prints whether each variant helped, made it worse, or did the same ([Measure lift against a baseline](docs/results/compare/lift.mdx)). - **Pinned environments**: an environment starts from a workspace, such as a git repository at a fixed ref, then adds files, setup steps, rules, skills, and MCP servers, so every attempt starts from the same state ([Define an environment](docs/configuration/environments/index.mdx)). - **History in Git**: recorded runs sit beside the code. `reeval query` reads them with SQL, and `reeval studio` shows each attempt's transcript, diff, and checks ([Record runs in Git](docs/evaluations/runs/recording.mdx)). ## Install ```bash npm i -D reeval ``` reeval runs on Node.js 22.13 or later. Real runs also need a sandbox runtime (Docker or Podman) and an API keys for the agents. `npx reeval doctor` checks both and names the fix for anything missing. ## Quick start Scaffold a config and one eval, then check that reeval loads them: ```bash npx reeval init --yes # reeval.config.ts on Cursor, and evals/greet.eval.ts npx reeval run --dry-run # smoke check: plans the run, no keys or sandbox ``` ```text REEVAL v0.16.1 dry run ➜ Run: … ➜ Harness: cursor:composer-2.5 (effort: not checked) ➜ Sandbox: docker ➜ Scope: 1 eval file · 1 eval · 1 cell · 1 attempt ➜ Est. Time: ~3m 00s (serial) eval harnesses variants runs attempts notes evals/greet.eval.ts › creates greet.js 1 — 1 1 effort: cursor:composer-2.5 not checked against Cursor's model catalog (CURSOR_API_KEY isn't set) … ``` Then build the agent's sandbox image, add its key, and run it for real: ```bash npx reeval image build echo 'CURSOR_API_KEY=…' >> .env.local npx reeval run ``` Without `--yes`, `init` asks which agent to configure. The starter eval only proves the setup works; the next section is what reeval is for. ## Measure whether a skill helps The Vercel AI SDK deprecates `generateObject` in favor of `Output.object`, and agents trained on older code still reach for the old call. This eval measures whether a skill shipped with the SDK fixes that. It runs the same task in two arms, four attempts each: `plain` has no skill, and `with-skill` adds the one in `skills/ai-sdk`. ```ts // evals/ai-sdk.eval.ts import { defineEnvironment, evaluate, expect } from 'reeval'; const plain = defineEnvironment('plain', {}); const withSkill = plain.extendEnvironment('with-skill', ({ skill }) => ({ skills: [skill.local('../skills/ai-sdk')], })); evaluate( 'triages a GitHub issue with the AI SDK', { runs: 4, compare: { control: plain, variants: [withSkill] } }, async (c) => { await c.send(` Add triage.ts: triageIssue(title, body) uses the ai package to return a typed { labels, priority, duplicateOf }. `); await expect(c, 'writes triage.ts').toHaveFile('triage.ts'); await expect(c, 'uses Output.object').toHaveFile('triage.ts', /Output\.object/); }, ); ``` ```text $ npx reeval run evals/ai-sdk.eval.ts … $ npx reeval show … pass triages a GitHub issue with the AI SDK arm pass plain 0 of 4 with-skill 4 of 4 helped check plain with-skill uses Output.object 0 of 4 4 of 4 expected triage.ts to match /Output\.object/ +1 more passed … 1 pass ``` Without the skill, no attempt used `Output.object`; with it, all four did. `helped` means the variant passed more often than the baseline, and `made it worse` less often. When the 95% interval of the difference spans zero (`-v` prints it), the `too-few-tries` remark names the `runs` that would make it hold. `+1 more passed` folds the checks that every arm passed. ## Run in CI and inspect runs | Command | Use it for | | --- | --- | | <code>reeval run --profile pr</code> | A run sized for a pull request. In the config `init` writes, that is one run per cell, no variants, and at most 20 attempts. | | <code>reeval compare --fail-on-regression</code> | Diff a run against its nearest recorded ancestor, and exit 1 when an eval regressed. | | <code>reeval show --md</code> | One run's results as Markdown, for a job summary or a PR comment. | | <code>reeval query</code> | Read-only SQL over every run, or built-in views such as `lift`, `effort`, and `failing_attempts`. | | <code>reeval studio</code> | Runs, transcripts, diffs, and checks in the browser, or a static site with `reeval studio build`. | Environments load extensions from where teams already keep them: - **Skills**: from a local folder, a git repository at a pinned ref, an npm package, the skills.sh registry, or written inline. - **MCP servers**: stdio, Streamable HTTP, or SSE servers, with their own environment variables and headers. - **Rules**: the project's instructions, written as `AGENTS.md`, or `CLAUDE.md` on Claude Code, before the first turn. ## Examples The recipes in [`docs/evaluations/recipes`](docs/evaluations/recipes) are complete evals for common jobs: - [Measure a skill](docs/evaluations/recipes/measure-a-skill.mdx): with and without a skill, gated on a minimum lift. - [Compare harnesses](docs/evaluations/recipes/compare-harnesses.mdx): one suite on several agents at once, with pass rate and effort per agent. - [Measure MCP servers](docs/evaluations/recipes/measure-mcp-servers.mdx): check the agent calls the server's tools, and whether answers improve. - [Check a recommendation](docs/evaluations/recipes/test-recommendations.mdx): ask which library to use, and let a judge decide whether the agent picks the right one. - [Fail a pull request](docs/evaluations/recipes/gate-pull-requests.mdx): run on every pull request and fail the check on a regression. The [`examples/`](examples) directory has 80 runnable projects, one idea each, from [`with-checks`](examples/with-checks) (deterministic checks) and [`with-skill-lift`](examples/with-skill-lift) (does a skill help?) to sandboxes, egress, judges, captures and real-world tasks. Most run on a fake agent with no keys or Docker; [`examples/README.md`](examples/README.md) lists them all. ## Documentation The manual in [`docs/`](docs/overview.mdx) covers harnesses, sandboxes, environments, secrets and egress, checks and judges, profiles, history, and every CLI command and flag. ## For coding agents To write or update evals, load [`skills/reeval/SKILL.md`](skills/reeval/SKILL.md). For every command, flag, and exit code, run `npx reeval --help`. To preview a run without starting sandboxes or spending tokens, run `npx reeval run --dry-run --json`. It prints the planned evals, cells, and attempts as one JSON document and exits 0. To check a machine's setup, run `npx reeval doctor --json`. It checks Node.js, the sandbox and its images, and the agents' keys, never prints a key, and changes nothing. To read past runs without calling a model, run `npx reeval query failing_attempts --json`, or `npx reeval analyze --context` for a run's digest. `npx reeval mcp` serves the run history over stdio as read-only MCP tools. ## License [Apache-2.0](LICENSE). Copyright 2026 Invertase Limited. The npm package ships third-party licenses and attributions in `THIRD_PARTY_NOTICES.md`. Questions about licensing: [oss@invertase.io](mailto:oss@invertase.io).

Chargement du README du dépôt…

Skills dans ce dépôt

classification en attente

skill

métier

description

mis à jour

reeval

non classé

How to write, run and debug evals with reeval, the eval runner for coding agents (npm package reeval; CLI `reeval`, alias `re`). Use whenever a task mentions reeval, reeval.config.ts, *.eval.ts files, `reeval run`, evals for a coding agent, comparing skills,…

Langue du texte source : anglais

2026-11-04

eval-rubric

non classé

The rubric reeval holds evals to, one section per remark kind. Load it when writing or reviewing a reeval eval's name, prompts, judge questions or compare, or when a reeval run prints a Remarks block.

Langue du texte source : anglais

2026-11-03

jsdoc

non classé

How this project documents its functions. Use whenever you add or change an exported function.

Langue du texte source : anglais

2026-11-02

setup

non classé

Read this first, before the reeval skill, whenever you're asked to set up, install, add, try or get started with reeval (the eval runner for coding agents), in a project or an empty directory, including "set up reeval here" and "add evals". It detects what's…

Langue du texte source : anglais

2026-11-02

add-a-language

non classé

How to add a new eval language runtime to reeval (Dart, Go, and so on): LanguageAdapter, stdio transport, attempt protocol v2, conformance goldens and a feature smoke fixture. Use when extending reeval beyond TypeScript *.eval.ts or wiring another…

Langue du texte source : anglais

2026-10-27

design

non classé

Acme's design guidelines for web pages. Use when building or styling any Acme page.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

The organization's greeting style. Use when writing greetings.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

The organization's greeting style. Use when writing greetings.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

The organization's greeting style. Use when writing greetings.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

The organization's greeting style. Use when writing greetings.

Langue du texte source : anglais

2026-10-27

house-farewell

non classé

The organization's farewell style. Use when signing off.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

The organization's greeting style. Use when writing greetings.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

The organization's greeting style. Use when writing greetings.

Langue du texte source : Indéterminée

2026-10-27

house-greeting

non classé

The house style for greetings in this organization. Use when writing any greeting function, message or text.

Langue du texte source : anglais

2026-10-27

house-greeting

non classé

Greet people in the house style.

Langue du texte source : anglais

2026-10-26

brand

non classé

Ledger's brand. Use when adding or styling buttons, links or headings in the Ledger site.

Langue du texte source : anglais

2026-10-24

testing

non classé

How this project tests its code. Use whenever you add or change a function in src/.

Langue du texte source : anglais

2026-10-24

errors

non classé

How this project's functions reject bad arguments. Use whenever you write a function that can be called with an argument it can't handle.

Langue du texte source : anglais

2026-10-24

project-layout

non classé

Where things are in this package and how to check a change. Use before you add or change any code here.

Langue du texte source : anglais

2026-10-24

release

non classé

How to prepare a release of this project. Use when asked to release, or to bump the version.

Langue du texte source : anglais

2026-10-24

changelog

non classé

How this project records changes. Use whenever you add, change or remove code.

Langue du texte source : anglais

2026-10-24

jsdoc

non classé

How this project documents its functions. Use whenever you add or change an exported function.

Langue du texte source : anglais

2026-10-24

jsdoc

non classé

How this project documents its code. Use whenever you add or change an exported function.

Langue du texte source : anglais

2026-10-24

house-style

non classé

The house style. Use when writing code here.

Langue du texte source : anglais

2026-10-21

house-style

non classé

The house style. Use when writing code here.

Langue du texte source : anglais

2026-10-21

plugin-probe

non classé

Runs the plugin probe. Use when asked to use the plugin-probe skill or to run the plugin probe.

Langue du texte source : anglais

2026-10-21

greet

non classé

Greets the user with the line the plugin's build wrote.

Langue du texte source : anglais

2026-10-21

greet

non classé

Greets.

Langue du texte source : Indéterminée

2026-10-21

review-lint

non classé

Lints a change the house way before review.

Langue du texte source : anglais

2026-10-21

tidy

non classé

Tidies imports.

Langue du texte source : Indéterminée

2026-10-21

review-lint

non classé

Lints a change the house way.

Langue du texte source : anglais

2026-10-21

lint

non classé

The project's own lint skill.

Langue du texte source : Indéterminée

2026-10-21

house-style

non classé

Use when writing greetings or user-facing messages in this codebase.

Langue du texte source : anglais

2026-10-18

greet

non classé

Greets people.

Langue du texte source : Indéterminée

2026-10-17

foo

non classé

A fixture skill whose path the fake agent cats.

Langue du texte source : anglais

2026-10-12

house-greeting

non classé

Greet people in the house style.

Langue du texte source : anglais

2026-09-28

contract-probe

non classé

Runs the contract probe. Use when asked to use the contract-probe skill or to run the contract probe.

Langue du texte source : anglais

2026-09-28

greeter

non classé

Greets people.

Langue du texte source : Indéterminée

2026-09-28

skill-reviewer

non classé

Reviews an agent skill before it ships: frontmatter, triggering and progressive disclosure. Use when: someone asks for a skill review.

Langue du texte source : anglais

2026-09-28

Affichage de 39 skills collectés sur 39.