invertase/reeval

Última actividad de origen registrada: 4 de noviembre de 2026 a las 19:51

Catálogo de SkillsMP actualizado: 5 de noviembre de 2026 a las 19:57

skills recopiladas: 39

Estrellas en GitHub: 13
Forks en GitHub: 0

Instalar con un asistente de IA

Copia este prompt en el asistente de IA que estés usando.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/invertase/reeval.

README del repositorio

Archivo de origen del README: README.md

Leer el README original

Mostrar README completo

``` ██ ▄▄ ▄▄▄ ▄▄▄▄ ▄▄▄▄ ▄▄ ▄▄ ▄▄▄▄ ██ ███▀▀▀ ██▄▄██ ██▄▄██ ██ ██ ▄▄▄██ ██ ██ ██ ▄ ██ ▄ ████ ██ ██ ██ ██ ▀▀▀▀ ▀▀▀▀ ▀▀ ▀▀▀▀▀ ▀▀ Evaluation runner for SDKs, CLIs, MCP servers, and docs. # Get started using the `/setup` skill: npx skills add invertase/reeval --skill setup # Or install manually: npm i -D reeval && npx reeval init ``` > [!IMPORTANT] > **Work in Progress**: reeval is under active development. Expect breaking changes between minor version releases. # reeval reeval is an open-source evaluation runner for developer tools. It gives a real coding agent a task in a fresh sandbox, then checks the result with assertions on files, commands, and the transcript, and with an LLM judge where an assertion can't decide. Run the same task with and without a skill, doc, or MCP server, and reeval reports whether the change helped. Evals are TypeScript files in `evals/`, next to the tool they test. Each run writes a JSON record, and recorded runs commit to Git, so `reeval compare` can diff a pull request's run against the last recorded one and fail the job when an eval regresses. ## Features Cursor ⋅ Claude Code ⋅ Codex ⋅ Antigravity ⋅ OpenCode ⋅ Goose ⋅ pi ⋅ Grok Build - **Real agents**: each harness runs the agent itself, with its own tools, skills, and rules, as it works on a developer's machine ([Choose an agent](docs/configuration/agents/index.mdx)). - **Keys stay on the host**: each attempt starts in a fresh sandbox that holds only a placeholder key. reeval's egress gateway swaps in the real key on requests to that provider and nowhere else ([Pass secrets](docs/configuration/environments/secrets.mdx)). - **Compare a change**: one eval runs a baseline and its variants with the same prompt and checks, then prints whether each variant helped, made it worse, or did the same ([Measure lift against a baseline](docs/results/compare/lift.mdx)). - **Pinned environments**: an environment starts from a workspace, such as a git repository at a fixed ref, then adds files, setup steps, rules, skills, and MCP servers, so every attempt starts from the same state ([Define an environment](docs/configuration/environments/index.mdx)). - **History in Git**: recorded runs sit beside the code. `reeval query` reads them with SQL, and `reeval studio` shows each attempt's transcript, diff, and checks ([Record runs in Git](docs/evaluations/runs/recording.mdx)). ## Install ```bash npm i -D reeval ``` reeval runs on Node.js 22.13 or later. Real runs also need a sandbox runtime (Docker or Podman) and an API keys for the agents. `npx reeval doctor` checks both and names the fix for anything missing. ## Quick start Scaffold a config and one eval, then check that reeval loads them: ```bash npx reeval init --yes # reeval.config.ts on Cursor, and evals/greet.eval.ts npx reeval run --dry-run # smoke check: plans the run, no keys or sandbox ``` ```text REEVAL v0.16.1 dry run ➜ Run: … ➜ Harness: cursor:composer-2.5 (effort: not checked) ➜ Sandbox: docker ➜ Scope: 1 eval file · 1 eval · 1 cell · 1 attempt ➜ Est. Time: ~3m 00s (serial) eval harnesses variants runs attempts notes evals/greet.eval.ts › creates greet.js 1 — 1 1 effort: cursor:composer-2.5 not checked against Cursor's model catalog (CURSOR_API_KEY isn't set) … ``` Then build the agent's sandbox image, add its key, and run it for real: ```bash npx reeval image build echo 'CURSOR_API_KEY=…' >> .env.local npx reeval run ``` Without `--yes`, `init` asks which agent to configure. The starter eval only proves the setup works; the next section is what reeval is for. ## Measure whether a skill helps The Vercel AI SDK deprecates `generateObject` in favor of `Output.object`, and agents trained on older code still reach for the old call. This eval measures whether a skill shipped with the SDK fixes that. It runs the same task in two arms, four attempts each: `plain` has no skill, and `with-skill` adds the one in `skills/ai-sdk`. ```ts // evals/ai-sdk.eval.ts import { defineEnvironment, evaluate, expect } from 'reeval'; const plain = defineEnvironment('plain', {}); const withSkill = plain.extendEnvironment('with-skill', ({ skill }) => ({ skills: [skill.local('../skills/ai-sdk')], })); evaluate( 'triages a GitHub issue with the AI SDK', { runs: 4, compare: { control: plain, variants: [withSkill] } }, async (c) => { await c.send(` Add triage.ts: triageIssue(title, body) uses the ai package to return a typed { labels, priority, duplicateOf }. `); await expect(c, 'writes triage.ts').toHaveFile('triage.ts'); await expect(c, 'uses Output.object').toHaveFile('triage.ts', /Output\.object/); }, ); ``` ```text $ npx reeval run evals/ai-sdk.eval.ts … $ npx reeval show … pass triages a GitHub issue with the AI SDK arm pass plain 0 of 4 with-skill 4 of 4 helped check plain with-skill uses Output.object 0 of 4 4 of 4 expected triage.ts to match /Output\.object/ +1 more passed … 1 pass ``` Without the skill, no attempt used `Output.object`; with it, all four did. `helped` means the variant passed more often than the baseline, and `made it worse` less often. When the 95% interval of the difference spans zero (`-v` prints it), the `too-few-tries` remark names the `runs` that would make it hold. `+1 more passed` folds the checks that every arm passed. ## Run in CI and inspect runs | Command | Use it for | | --- | --- | | <code>reeval run --profile pr</code> | A run sized for a pull request. In the config `init` writes, that is one run per cell, no variants, and at most 20 attempts. | | <code>reeval compare --fail-on-regression</code> | Diff a run against its nearest recorded ancestor, and exit 1 when an eval regressed. | | <code>reeval show --md</code> | One run's results as Markdown, for a job summary or a PR comment. | | <code>reeval query</code> | Read-only SQL over every run, or built-in views such as `lift`, `effort`, and `failing_attempts`. | | <code>reeval studio</code> | Runs, transcripts, diffs, and checks in the browser, or a static site with `reeval studio build`. | Environments load extensions from where teams already keep them: - **Skills**: from a local folder, a git repository at a pinned ref, an npm package, the skills.sh registry, or written inline. - **MCP servers**: stdio, Streamable HTTP, or SSE servers, with their own environment variables and headers. - **Rules**: the project's instructions, written as `AGENTS.md`, or `CLAUDE.md` on Claude Code, before the first turn. ## Examples The recipes in [`docs/evaluations/recipes`](docs/evaluations/recipes) are complete evals for common jobs: - [Measure a skill](docs/evaluations/recipes/measure-a-skill.mdx): with and without a skill, gated on a minimum lift. - [Compare harnesses](docs/evaluations/recipes/compare-harnesses.mdx): one suite on several agents at once, with pass rate and effort per agent. - [Measure MCP servers](docs/evaluations/recipes/measure-mcp-servers.mdx): check the agent calls the server's tools, and whether answers improve. - [Check a recommendation](docs/evaluations/recipes/test-recommendations.mdx): ask which library to use, and let a judge decide whether the agent picks the right one. - [Fail a pull request](docs/evaluations/recipes/gate-pull-requests.mdx): run on every pull request and fail the check on a regression. The [`examples/`](examples) directory has 80 runnable projects, one idea each, from [`with-checks`](examples/with-checks) (deterministic checks) and [`with-skill-lift`](examples/with-skill-lift) (does a skill help?) to sandboxes, egress, judges, captures and real-world tasks. Most run on a fake agent with no keys or Docker; [`examples/README.md`](examples/README.md) lists them all. ## Documentation The manual in [`docs/`](docs/overview.mdx) covers harnesses, sandboxes, environments, secrets and egress, checks and judges, profiles, history, and every CLI command and flag. ## For coding agents To write or update evals, load [`skills/reeval/SKILL.md`](skills/reeval/SKILL.md). For every command, flag, and exit code, run `npx reeval --help`. To preview a run without starting sandboxes or spending tokens, run `npx reeval run --dry-run --json`. It prints the planned evals, cells, and attempts as one JSON document and exits 0. To check a machine's setup, run `npx reeval doctor --json`. It checks Node.js, the sandbox and its images, and the agents' keys, never prints a key, and changes nothing. To read past runs without calling a model, run `npx reeval query failing_attempts --json`, or `npx reeval analyze --context` for a run's digest. `npx reeval mcp` serves the run history over stdio as read-only MCP tools. ## License [Apache-2.0](LICENSE). Copyright 2026 Invertase Limited. The npm package ships third-party licenses and attributions in `THIRD_PARTY_NOTICES.md`. Questions about licensing: [oss@invertase.io](mailto:oss@invertase.io).

Cargando el README del repositorio…

Skills en este repositorio

clasificación pendiente

skill

ocupación

descripción

actualizado

reeval

sin clasificar

How to write, run and debug evals with reeval, the eval runner for coding agents (npm package reeval; CLI `reeval`, alias `re`). Use whenever a task mentions reeval, reeval.config.ts, *.eval.ts files, `reeval run`, evals for a coding agent, comparing skills,…

Idioma del texto original: inglés

2026-11-04

eval-rubric

sin clasificar

The rubric reeval holds evals to, one section per remark kind. Load it when writing or reviewing a reeval eval's name, prompts, judge questions or compare, or when a reeval run prints a Remarks block.

Idioma del texto original: inglés

2026-11-03

jsdoc

sin clasificar

How this project documents its functions. Use whenever you add or change an exported function.

Idioma del texto original: inglés

2026-11-02

setup

sin clasificar

Read this first, before the reeval skill, whenever you're asked to set up, install, add, try or get started with reeval (the eval runner for coding agents), in a project or an empty directory, including "set up reeval here" and "add evals". It detects what's…

Idioma del texto original: inglés

2026-11-02

add-a-language

sin clasificar

How to add a new eval language runtime to reeval (Dart, Go, and so on): LanguageAdapter, stdio transport, attempt protocol v2, conformance goldens and a feature smoke fixture. Use when extending reeval beyond TypeScript *.eval.ts or wiring another…

Idioma del texto original: inglés

2026-10-27

design

sin clasificar

Acme's design guidelines for web pages. Use when building or styling any Acme page.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

The organization's greeting style. Use when writing greetings.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

The organization's greeting style. Use when writing greetings.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

The organization's greeting style. Use when writing greetings.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

The organization's greeting style. Use when writing greetings.

Idioma del texto original: inglés

2026-10-27

house-farewell

sin clasificar

The organization's farewell style. Use when signing off.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

The organization's greeting style. Use when writing greetings.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

The organization's greeting style. Use when writing greetings.

Idioma del texto original: Sin determinar

2026-10-27

house-greeting

sin clasificar

The house style for greetings in this organization. Use when writing any greeting function, message or text.

Idioma del texto original: inglés

2026-10-27

house-greeting

sin clasificar

Greet people in the house style.

Idioma del texto original: inglés

2026-10-26

brand

sin clasificar

Ledger's brand. Use when adding or styling buttons, links or headings in the Ledger site.

Idioma del texto original: inglés

2026-10-24

testing

sin clasificar

How this project tests its code. Use whenever you add or change a function in src/.

Idioma del texto original: inglés

2026-10-24

errors

sin clasificar

How this project's functions reject bad arguments. Use whenever you write a function that can be called with an argument it can't handle.

Idioma del texto original: inglés

2026-10-24

project-layout

sin clasificar

Where things are in this package and how to check a change. Use before you add or change any code here.

Idioma del texto original: inglés

2026-10-24

release

sin clasificar

How to prepare a release of this project. Use when asked to release, or to bump the version.

Idioma del texto original: inglés

2026-10-24

changelog

sin clasificar

How this project records changes. Use whenever you add, change or remove code.

Idioma del texto original: inglés

2026-10-24

jsdoc

sin clasificar

How this project documents its functions. Use whenever you add or change an exported function.

Idioma del texto original: inglés

2026-10-24

jsdoc

sin clasificar

How this project documents its code. Use whenever you add or change an exported function.

Idioma del texto original: inglés

2026-10-24

house-style

sin clasificar

The house style. Use when writing code here.

Idioma del texto original: inglés

2026-10-21

house-style

sin clasificar

The house style. Use when writing code here.

Idioma del texto original: inglés

2026-10-21

plugin-probe

sin clasificar

Runs the plugin probe. Use when asked to use the plugin-probe skill or to run the plugin probe.

Idioma del texto original: inglés

2026-10-21

greet

sin clasificar

Greets the user with the line the plugin's build wrote.

Idioma del texto original: inglés

2026-10-21

greet

sin clasificar

Greets.

Idioma del texto original: Sin determinar

2026-10-21

review-lint

sin clasificar

Lints a change the house way before review.

Idioma del texto original: inglés

2026-10-21

tidy

sin clasificar

Tidies imports.

Idioma del texto original: Sin determinar

2026-10-21

review-lint

sin clasificar

Lints a change the house way.

Idioma del texto original: inglés

2026-10-21

lint

sin clasificar

The project's own lint skill.

Idioma del texto original: Sin determinar

2026-10-21

house-style

sin clasificar

Use when writing greetings or user-facing messages in this codebase.

Idioma del texto original: inglés

2026-10-18

greet

sin clasificar

Greets people.

Idioma del texto original: Sin determinar

2026-10-17

foo

sin clasificar

A fixture skill whose path the fake agent cats.

Idioma del texto original: inglés

2026-10-12

house-greeting

sin clasificar

Greet people in the house style.

Idioma del texto original: inglés

2026-09-28

contract-probe

sin clasificar

Runs the contract probe. Use when asked to use the contract-probe skill or to run the contract probe.

Idioma del texto original: inglés

2026-09-28

greeter

sin clasificar

Greets people.

Idioma del texto original: Sin determinar

2026-09-28

skill-reviewer

sin clasificar

Reviews an agent skill before it ships: frontmatter, triggering and progressive disclosure. Use when: someone asks for a skill review.

Idioma del texto original: inglés

2026-09-28

Mostrando 39 de 39 skills recopiladas.