invertase/reeval

Última atividade de origem registrada: 4 de novembro de 2026 às 19:51

Catálogo do SkillsMP atualizado: 5 de novembro de 2026 às 19:57

skills coletadas: 39

Estrelas no GitHub: 13
Forks no GitHub: 0

Instalar com um assistente de IA

Copie este prompt para o assistente de IA que você usa.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/invertase/reeval.

README do repositório

Arquivo de origem do README: README.md

Ler o README original

Expandir README completo

``` ██ ▄▄ ▄▄▄ ▄▄▄▄ ▄▄▄▄ ▄▄ ▄▄ ▄▄▄▄ ██ ███▀▀▀ ██▄▄██ ██▄▄██ ██ ██ ▄▄▄██ ██ ██ ██ ▄ ██ ▄ ████ ██ ██ ██ ██ ▀▀▀▀ ▀▀▀▀ ▀▀ ▀▀▀▀▀ ▀▀ Evaluation runner for SDKs, CLIs, MCP servers, and docs. # Get started using the `/setup` skill: npx skills add invertase/reeval --skill setup # Or install manually: npm i -D reeval && npx reeval init ``` > [!IMPORTANT] > **Work in Progress**: reeval is under active development. Expect breaking changes between minor version releases. # reeval reeval is an open-source evaluation runner for developer tools. It gives a real coding agent a task in a fresh sandbox, then checks the result with assertions on files, commands, and the transcript, and with an LLM judge where an assertion can't decide. Run the same task with and without a skill, doc, or MCP server, and reeval reports whether the change helped. Evals are TypeScript files in `evals/`, next to the tool they test. Each run writes a JSON record, and recorded runs commit to Git, so `reeval compare` can diff a pull request's run against the last recorded one and fail the job when an eval regresses. ## Features Cursor ⋅ Claude Code ⋅ Codex ⋅ Antigravity ⋅ OpenCode ⋅ Goose ⋅ pi ⋅ Grok Build - **Real agents**: each harness runs the agent itself, with its own tools, skills, and rules, as it works on a developer's machine ([Choose an agent](docs/configuration/agents/index.mdx)). - **Keys stay on the host**: each attempt starts in a fresh sandbox that holds only a placeholder key. reeval's egress gateway swaps in the real key on requests to that provider and nowhere else ([Pass secrets](docs/configuration/environments/secrets.mdx)). - **Compare a change**: one eval runs a baseline and its variants with the same prompt and checks, then prints whether each variant helped, made it worse, or did the same ([Measure lift against a baseline](docs/results/compare/lift.mdx)). - **Pinned environments**: an environment starts from a workspace, such as a git repository at a fixed ref, then adds files, setup steps, rules, skills, and MCP servers, so every attempt starts from the same state ([Define an environment](docs/configuration/environments/index.mdx)). - **History in Git**: recorded runs sit beside the code. `reeval query` reads them with SQL, and `reeval studio` shows each attempt's transcript, diff, and checks ([Record runs in Git](docs/evaluations/runs/recording.mdx)). ## Install ```bash npm i -D reeval ``` reeval runs on Node.js 22.13 or later. Real runs also need a sandbox runtime (Docker or Podman) and an API keys for the agents. `npx reeval doctor` checks both and names the fix for anything missing. ## Quick start Scaffold a config and one eval, then check that reeval loads them: ```bash npx reeval init --yes # reeval.config.ts on Cursor, and evals/greet.eval.ts npx reeval run --dry-run # smoke check: plans the run, no keys or sandbox ``` ```text REEVAL v0.16.1 dry run ➜ Run: … ➜ Harness: cursor:composer-2.5 (effort: not checked) ➜ Sandbox: docker ➜ Scope: 1 eval file · 1 eval · 1 cell · 1 attempt ➜ Est. Time: ~3m 00s (serial) eval harnesses variants runs attempts notes evals/greet.eval.ts › creates greet.js 1 — 1 1 effort: cursor:composer-2.5 not checked against Cursor's model catalog (CURSOR_API_KEY isn't set) … ``` Then build the agent's sandbox image, add its key, and run it for real: ```bash npx reeval image build echo 'CURSOR_API_KEY=…' >> .env.local npx reeval run ``` Without `--yes`, `init` asks which agent to configure. The starter eval only proves the setup works; the next section is what reeval is for. ## Measure whether a skill helps The Vercel AI SDK deprecates `generateObject` in favor of `Output.object`, and agents trained on older code still reach for the old call. This eval measures whether a skill shipped with the SDK fixes that. It runs the same task in two arms, four attempts each: `plain` has no skill, and `with-skill` adds the one in `skills/ai-sdk`. ```ts // evals/ai-sdk.eval.ts import { defineEnvironment, evaluate, expect } from 'reeval'; const plain = defineEnvironment('plain', {}); const withSkill = plain.extendEnvironment('with-skill', ({ skill }) => ({ skills: [skill.local('../skills/ai-sdk')], })); evaluate( 'triages a GitHub issue with the AI SDK', { runs: 4, compare: { control: plain, variants: [withSkill] } }, async (c) => { await c.send(` Add triage.ts: triageIssue(title, body) uses the ai package to return a typed { labels, priority, duplicateOf }. `); await expect(c, 'writes triage.ts').toHaveFile('triage.ts'); await expect(c, 'uses Output.object').toHaveFile('triage.ts', /Output\.object/); }, ); ``` ```text $ npx reeval run evals/ai-sdk.eval.ts … $ npx reeval show … pass triages a GitHub issue with the AI SDK arm pass plain 0 of 4 with-skill 4 of 4 helped check plain with-skill uses Output.object 0 of 4 4 of 4 expected triage.ts to match /Output\.object/ +1 more passed … 1 pass ``` Without the skill, no attempt used `Output.object`; with it, all four did. `helped` means the variant passed more often than the baseline, and `made it worse` less often. When the 95% interval of the difference spans zero (`-v` prints it), the `too-few-tries` remark names the `runs` that would make it hold. `+1 more passed` folds the checks that every arm passed. ## Run in CI and inspect runs | Command | Use it for | | --- | --- | | <code>reeval run --profile pr</code> | A run sized for a pull request. In the config `init` writes, that is one run per cell, no variants, and at most 20 attempts. | | <code>reeval compare --fail-on-regression</code> | Diff a run against its nearest recorded ancestor, and exit 1 when an eval regressed. | | <code>reeval show --md</code> | One run's results as Markdown, for a job summary or a PR comment. | | <code>reeval query</code> | Read-only SQL over every run, or built-in views such as `lift`, `effort`, and `failing_attempts`. | | <code>reeval studio</code> | Runs, transcripts, diffs, and checks in the browser, or a static site with `reeval studio build`. | Environments load extensions from where teams already keep them: - **Skills**: from a local folder, a git repository at a pinned ref, an npm package, the skills.sh registry, or written inline. - **MCP servers**: stdio, Streamable HTTP, or SSE servers, with their own environment variables and headers. - **Rules**: the project's instructions, written as `AGENTS.md`, or `CLAUDE.md` on Claude Code, before the first turn. ## Examples The recipes in [`docs/evaluations/recipes`](docs/evaluations/recipes) are complete evals for common jobs: - [Measure a skill](docs/evaluations/recipes/measure-a-skill.mdx): with and without a skill, gated on a minimum lift. - [Compare harnesses](docs/evaluations/recipes/compare-harnesses.mdx): one suite on several agents at once, with pass rate and effort per agent. - [Measure MCP servers](docs/evaluations/recipes/measure-mcp-servers.mdx): check the agent calls the server's tools, and whether answers improve. - [Check a recommendation](docs/evaluations/recipes/test-recommendations.mdx): ask which library to use, and let a judge decide whether the agent picks the right one. - [Fail a pull request](docs/evaluations/recipes/gate-pull-requests.mdx): run on every pull request and fail the check on a regression. The [`examples/`](examples) directory has 80 runnable projects, one idea each, from [`with-checks`](examples/with-checks) (deterministic checks) and [`with-skill-lift`](examples/with-skill-lift) (does a skill help?) to sandboxes, egress, judges, captures and real-world tasks. Most run on a fake agent with no keys or Docker; [`examples/README.md`](examples/README.md) lists them all. ## Documentation The manual in [`docs/`](docs/overview.mdx) covers harnesses, sandboxes, environments, secrets and egress, checks and judges, profiles, history, and every CLI command and flag. ## For coding agents To write or update evals, load [`skills/reeval/SKILL.md`](skills/reeval/SKILL.md). For every command, flag, and exit code, run `npx reeval --help`. To preview a run without starting sandboxes or spending tokens, run `npx reeval run --dry-run --json`. It prints the planned evals, cells, and attempts as one JSON document and exits 0. To check a machine's setup, run `npx reeval doctor --json`. It checks Node.js, the sandbox and its images, and the agents' keys, never prints a key, and changes nothing. To read past runs without calling a model, run `npx reeval query failing_attempts --json`, or `npx reeval analyze --context` for a run's digest. `npx reeval mcp` serves the run history over stdio as read-only MCP tools. ## License [Apache-2.0](LICENSE). Copyright 2026 Invertase Limited. The npm package ships third-party licenses and attributions in `THIRD_PARTY_NOTICES.md`. Questions about licensing: [oss@invertase.io](mailto:oss@invertase.io).

Carregando o README do repositório…

Skills neste repositório

classificação pendente

skill

ocupação

descrição

atualizado

reeval

sem classificação

How to write, run and debug evals with reeval, the eval runner for coding agents (npm package reeval; CLI `reeval`, alias `re`). Use whenever a task mentions reeval, reeval.config.ts, *.eval.ts files, `reeval run`, evals for a coding agent, comparing skills,…

Idioma do texto original: inglês

2026-11-04

eval-rubric

sem classificação

The rubric reeval holds evals to, one section per remark kind. Load it when writing or reviewing a reeval eval's name, prompts, judge questions or compare, or when a reeval run prints a Remarks block.

Idioma do texto original: inglês

2026-11-03

jsdoc

sem classificação

How this project documents its functions. Use whenever you add or change an exported function.

Idioma do texto original: inglês

2026-11-02

setup

sem classificação

Read this first, before the reeval skill, whenever you're asked to set up, install, add, try or get started with reeval (the eval runner for coding agents), in a project or an empty directory, including "set up reeval here" and "add evals". It detects what's…

Idioma do texto original: inglês

2026-11-02

add-a-language

sem classificação

How to add a new eval language runtime to reeval (Dart, Go, and so on): LanguageAdapter, stdio transport, attempt protocol v2, conformance goldens and a feature smoke fixture. Use when extending reeval beyond TypeScript *.eval.ts or wiring another…

Idioma do texto original: inglês

2026-10-27

design

sem classificação

Acme's design guidelines for web pages. Use when building or styling any Acme page.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

The organization's greeting style. Use when writing greetings.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

The organization's greeting style. Use when writing greetings.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

The organization's greeting style. Use when writing greetings.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

The organization's greeting style. Use when writing greetings.

Idioma do texto original: inglês

2026-10-27

house-farewell

sem classificação

The organization's farewell style. Use when signing off.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

The organization's greeting style. Use when writing greetings.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

The organization's greeting style. Use when writing greetings.

Idioma do texto original: Indeterminado

2026-10-27

house-greeting

sem classificação

The house style for greetings in this organization. Use when writing any greeting function, message or text.

Idioma do texto original: inglês

2026-10-27

house-greeting

sem classificação

Greet people in the house style.

Idioma do texto original: inglês

2026-10-26

brand

sem classificação

Ledger's brand. Use when adding or styling buttons, links or headings in the Ledger site.

Idioma do texto original: inglês

2026-10-24

testing

sem classificação

How this project tests its code. Use whenever you add or change a function in src/.

Idioma do texto original: inglês

2026-10-24

errors

sem classificação

How this project's functions reject bad arguments. Use whenever you write a function that can be called with an argument it can't handle.

Idioma do texto original: inglês

2026-10-24

project-layout

sem classificação

Where things are in this package and how to check a change. Use before you add or change any code here.

Idioma do texto original: inglês

2026-10-24

release

sem classificação

How to prepare a release of this project. Use when asked to release, or to bump the version.

Idioma do texto original: inglês

2026-10-24

changelog

sem classificação

How this project records changes. Use whenever you add, change or remove code.

Idioma do texto original: inglês

2026-10-24

jsdoc

sem classificação

How this project documents its functions. Use whenever you add or change an exported function.

Idioma do texto original: inglês

2026-10-24

jsdoc

sem classificação

How this project documents its code. Use whenever you add or change an exported function.

Idioma do texto original: inglês

2026-10-24

house-style

sem classificação

The house style. Use when writing code here.

Idioma do texto original: inglês

2026-10-21

house-style

sem classificação

The house style. Use when writing code here.

Idioma do texto original: inglês

2026-10-21

plugin-probe

sem classificação

Runs the plugin probe. Use when asked to use the plugin-probe skill or to run the plugin probe.

Idioma do texto original: inglês

2026-10-21

greet

sem classificação

Greets the user with the line the plugin's build wrote.

Idioma do texto original: inglês

2026-10-21

greet

sem classificação

Greets.

Idioma do texto original: Indeterminado

2026-10-21

review-lint

sem classificação

Lints a change the house way before review.

Idioma do texto original: inglês

2026-10-21

tidy

sem classificação

Tidies imports.

Idioma do texto original: Indeterminado

2026-10-21

review-lint

sem classificação

Lints a change the house way.

Idioma do texto original: inglês

2026-10-21

lint

sem classificação

The project's own lint skill.

Idioma do texto original: Indeterminado

2026-10-21

house-style

sem classificação

Use when writing greetings or user-facing messages in this codebase.

Idioma do texto original: inglês

2026-10-18

greet

sem classificação

Greets people.

Idioma do texto original: Indeterminado

2026-10-17

foo

sem classificação

A fixture skill whose path the fake agent cats.

Idioma do texto original: inglês

2026-10-12

house-greeting

sem classificação

Greet people in the house style.

Idioma do texto original: inglês

2026-09-28

contract-probe

sem classificação

Runs the contract probe. Use when asked to use the contract-probe skill or to run the contract probe.

Idioma do texto original: inglês

2026-09-28

greeter

sem classificação

Greets people.

Idioma do texto original: Indeterminado

2026-09-28

skill-reviewer

sem classificação

Reviews an agent skill before it ships: frontmatter, triggering and progressive disclosure. Use when: someone asks for a skill review.

Idioma do texto original: inglês

2026-09-28

Mostrando 39 de 39 skills coletadas.