benchflow-ai/benchflow

ملف benchflow-ai

آخر نشاط مصدر مسجل: ٢٠ سبتمبر ٢٠٢٦ في ٠٥:٣٧

آخر تحديث لفهرس SkillsMP: ٢٦ أكتوبر ٢٠٢٦ في ١١:٣٧

skills مجمعة: ١٥

نجوم GitHub: ٣٥٥
تفرعات GitHub: ٧٤

ملف README للمستودع

مصدر ملف README للمستودع: README.md

قراءة الملف الأصلي

عرض ملف README كاملًا

<div align="center"> <h1>BenchFlow</h1> <p>The universal environment framework — a benchmark is just a frozen environment.</p> <a href="https://pypi.org/project/benchflow/" target="_blank"> <img src="https://img.shields.io/badge/PyPI-benchflow-3775A9?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI package"> </a> <a href="https://discord.gg/mZ9Rc8q8W3" target="_blank"> <img src="https://img.shields.io/badge/Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Discord"> </a> </div> ## What BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. **A benchmark is just a frozen environment** — point BenchFlow at any of them, drive it with *any* ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle. ## Quick start: 1. Submit a trajectory **Send these to your coding agent.** Copy the block below and paste it as a message to Claude Code, Codex, OpenCode, or Cursor. That is the $2,000 eval prize path. No BenchFlow account, API key, or Azure login. > Submit my relevant local Claude Code, Codex, OpenCode, or Cursor session from the re:Agent e2e agentic science / ai4bio hackathon (last 72 hours, on this laptop) to the BenchFlow eval prize. > > 1. First make sure the latest benchflow CLI is installed: uv tool install --python 3.12 --upgrade --force benchflow > > 2. Then read https://raw.githubusercontent.com/benchflow-ai/benchflow/main/.agents/skills/benchflow-traj-upload/SKILL.md and follow it: find a session, open the viewer, and upload only after I (the human) review it. The agent finds sessions on this machine, opens the viewer, and uploads after you like what you see. The prompt has the agent install the latest BenchFlow first (`uv tool install --python 3.12 --upgrade --force benchflow`). The `bench traj` commands also print a one-line upgrade hint when a newer release is available. Prefer the terminal instead? The guided upload inspects before anything leaves your machine — it renders a redacted trajectory report (step counts, masked secrets, preview) and asks for confirmation: ```bash uv tool install --python 3.12 --upgrade benchflow bench traj upload ``` Detected secret values are replaced locally with `<XXX-benchflow-key-values-XXX>` before upload, and the full redacted report is retained in the uploaded `manifest.json`. See the [upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md) or the [trajectory upload guide](./docs/traj-upload.md). Optional — set the skill up once, then keep talking to the agent: ```bash npx skills add benchflow-ai/benchflow --skill benchflow-traj-upload # or, if BenchFlow is already installed: bench traj setup ``` `npx skills add` asks which agents to install for. `bench traj setup` copies the skill into this project and prints the same agent prompt. See the [upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md). ## Quick start: 2. Run with a ChatGPT or Claude subscription No OpenAI or Anthropic API key is required. Start Docker, install BenchFlow, then run **one** of these options. BenchFlow detects the saved host login and makes it available to the agent inside the sandbox. ```bash uv tool install --python 3.12 --upgrade benchflow docker info >/dev/null # Docker must be running ``` ### ChatGPT subscription via Codex Install the [Codex CLI](https://github.com/openai/codex), then: ```bash codex login unset OPENAI_API_KEY CODEX_API_KEY # ensure subscription auth is used bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/citation-check \ --agent codex \ --model gpt-5.5 \ --sandbox docker ``` ### Claude subscription via Claude Code Install [Claude Code](https://code.claude.com/docs/en/quickstart), then: ```bash claude auth login unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN # ensure subscription auth is used bench eval run \ --source-repo benchflow-ai/skillsbench \ --source-path tasks/citation-check \ --agent claude \ --model claude-sonnet-4-6 \ --sandbox docker ``` The agent may pass or fail the benchmark task; either result means the evaluation completed. Each run writes rewards, token usage, and the full trajectory under `jobs/`. See [Getting started](./docs/getting-started.md) for other agents, models, and sandboxes. ## Install Install or upgrade to the latest stable release from PyPI with `uv`: ```bash uv tool install --python 3.12 --upgrade benchflow ``` - Confirm with `bench --version`. - BenchFlow CLI releases require Python 3.12 or newer. Keep `--python 3.12` in the install command so `uv` does not resolve an older Python-compatible package that lacks the CLI entrypoints. - If you see `Executables already exist: bench, benchflow`, re-run with `uv tool install --python 3.12 --upgrade --force benchflow` to replace stale entrypoints from an older install. - For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`. Internal users wanting the newest preview from `main` install the [internal preview channel](./docs/release.md) (`uv tool install --python 3.12 --prerelease allow --upgrade benchflow`). **Requirements & auth.** Install [uv](https://docs.astral.sh/uv/); the `--python 3.12` flag lets it provision a compatible interpreter for the tool install. Set `DAYTONA_API_KEY` for Daytona or configure Modal auth for Modal; export an agent API key (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, …) or use subscription auth (`claude auth login` / `codex login`). Provider-prefixed models may need provider-specific credentials; Azure Foundry uses `AZURE_API_KEY` + `AZURE_API_ENDPOINT`. ## Documentation Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/concepts.md) for the mental model. Prefer to have an AI coding agent run the whole quickstart for you? Paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code, Codex CLI, or Gemini CLI. Then by goal: | If you want to… | Read | |------------------|------| | Run an eval on an existing task | [Getting started](./docs/getting-started.md) | | Understand how BenchFlow runs *any* benchmark (the three-layer model) | [Run any benchmark](./docs/running-any-benchmark.md) | | Have an AI agent install + run the quickstart end to end | [Agent quickstart prompt](./docs/agent-quickstart.md) | | Run an agent from the public agents repo (goose, qwen-code, prime-agent, …) | [Running external agents](./docs/external-agents.md) | | Understand Rollout / Scene / Role / Verifier | [Concepts](./docs/concepts.md) | | Author a new task | [Task authoring](./docs/task-authoring.md) | | Author a task in the native `task.md` format | [Native task.md authoring](./docs/task-authoring-task-md.md) | | Run a hosted PrimeIntellect / Verifiers environment | [CLI reference](./docs/reference/cli.md) | | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) | | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) | | Skill evaluation (when the artifact is a skill, not a workspace) | [Skill eval](./docs/skill-eval.md) | | Contribute a trajectory capture | [Trajectory upload](./docs/traj-upload.md) | | Understand the security model | [Sandbox hardening](./docs/sandbox-hardening.md) | | Use public vs internal preview SDK releases | [Release channels](./docs/release.md) | | CLI flags + commands | [CLI reference](./docs/reference/cli.md) | | Python API surface | [Python API reference](./docs/reference/python-api.md) | Notebooks and runnable example scripts live under [`docs/examples/`](./docs/examples/) so examples stay versioned with the docs that explain them. > **`bench agent` vs `bench eval adopt`.** `bench agent list` / `bench agent show` > inspect **registered AI agents** (the solver programs like Claude Code or > Gemini CLI). Onboarding a third-party benchmark into `benchmarks/<name>/` is a > separate workflow — `bench eval adopt <source>` scaffolds and drives the > conversion, and `bench eval adopt <name> --verify` parity-gates it. (The legacy > `bench agent create|run|verify` commands still work as deprecated aliases.) > See the [CLI reference](./docs/reference/cli.md#bench-eval-adopt) for details. ## Benchmark task sources Benchmark datasets live in external Git repos and are referenced with two fields: ```yaml # benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml source: repo: benchflow-ai/benchmarks # GitHub org/repo path: datasets/harvey-lab/tasks # optional subpath within repo ref: main # optional branch/tag agent: gemini model: gemini/gemini-3.1-flash-lite-preview ``` Run any benchmark via the CLI: ```bash # From a YAML config (shipped with the repo) bench eval run --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml # Inline — mirrors the YAML source fields bench eval run \ --source-repo benchflow-ai/skillsbench --source-path tasks \ --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64 ``` Repos are cloned and cached locally under `.cache/datasets/` on first use. Hosted environments are another source type. Instead of a repo, pass `--source-env` with the environment's pinned source version to run an external PrimeIntellect / Verifiers environment on its own native harness — BenchFlow preserves the hosted identity (`env_uid`, `hub_url`) and still writes the shared rollout output contract. See the [CLI reference](./docs/reference/cli.md) for the full hosted-environment command shape. Downstream projects should depend on the public PyPI release by default. For internal validation before the next public release, install or lock the internal preview channel with prereleases enabled; see [Release channels](./docs/release.md). ## Authoring tasks A task is one `task.md` (YAML frontmatter for config + a markdown prompt body) plus `environment/` and `verifier/` sidecars. The `bench tasks` commands cover the authoring lifecycle: ```bash bench tasks init my-task # scaffold a task.md package under tasks/ bench tasks check tasks/my-task # validate (default --level structural) bench tasks migrate legacy-task/ --remove-legacy # convert old split packages to task.md bench tasks export tasks/my-task out/ # write a compatibility export + loss report ``` See [Native task.md authoring](./docs/task-authoring-task-md.md) and the [task standard](./docs/task-standard.md). ## Featured - **Progressive disclosure on SWE-bench Pro** — the `BaseUser` abstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at [`docs/examples/swebench_pro_progressive_disclosure.ipynb`](./docs/examples/swebench_pro_progressive_disclosure.ipynb). See [Progressive disclosure](./docs/progressive-disclosure.md). ## Audience - **Eval researchers / paper writers** → [Getting started](./docs/getting-started.md) → [Concepts](./docs/concepts.md) → [Use cases](./docs/use-cases.md) - **Task authors** → [Task authoring](./docs/task-authoring.md) → [Sandbox hardening](./docs/sandbox-hardening.md) - **Agent builders integrating with benchflow** → [Concepts](./docs/concepts.md) → [Python API reference](./docs/reference/python-api.md) → [`benchflow.agents.registry`](./src/benchflow/agents/registry.py) - **External benchmark adapters** → [Task authoring](./docs/task-authoring.md) → [Progressive disclosure](./docs/progressive-disclosure.md#comparison-with-multi-agent-simulated-user) ## Contributing PRs welcome. Open against `main`. CI runs ruff + tests on every PR; please run `ruff check .` and `pytest tests/` locally first. Release channels are documented in [Release channels](./docs/release.md). In short: merges to `main` publish an internal preview after CI passes, while a matching release tag publishes the public release. ## License Apache-2.0.

جارٍ تحميل ملف README للمستودع…

Skills في هذا المستودع

المهن الحاسوبية الأخرى مطوّرو البرمجيات محللو ضمان جودة البرمجيات والمختبرون تقنيو الهندسة (باستثناء رسامي المخططات)، جميع الآخرون

٦ فئات مهنية · 100% مصنفة

skill

المهنة

الوصف

آخر تحديث

benchflow

المهن الحاسوبية الأخرى

Run agent benchmarks, create tasks, analyze results, and manage agents using BenchFlow. Use when asked to benchmark an AI coding agent, run a benchmark suite, create tasks, view trajectories, or compare agent performance.

لغة النص الأصلي: الإنجليزية

2026-09-20

benchflow-experiment-review

محللو ضمان جودة البرمجيات والمختبرون

Review Benchflow or SkillsBench task-run trajectories and integration-test Benchflow code changes. Use this skill whenever the user asks to audit traj health, failed or timed-out runs, healthy pass/fail/timeout status, no-skill leakage, skill loading, reward…

لغة النص الأصلي: الإنجليزية

2026-08-22

benchflow-traj-upload-ops

مطوّرو البرمجيات

Operate, test, troubleshoot, and explain `bench traj upload` for public or trusted-direct trajectory contributions, including interactive and fully specified commands, dry runs, input validation, local secret masking, trajectory reports and previews, manifest…

لغة النص الأصلي: الإنجليزية

2026-08-17

benchflow-traj-upload

المهن الحاسوبية الأخرى

Find a local Claude Code or Codex session, open the BenchFlow trajectory viewer, and submit it after the user reviews it. Use this skill whenever someone pastes a BenchFlow eval prize line, wants to submit / share / contribute / upload a trajectory, set up…

لغة النص الأصلي: الإنجليزية

2026-08-16

task-creator

المهن الحاسوبية الأخرى

SkillsBench task authoring — walk a contributor from idea to submission-ready task following CONTRIBUTING.md and the task-implementation rubric. Use when the user wants to create a new SkillsBench task, scaffold a task from an existing workflow (notebook,…

لغة النص الأصلي: الإنجليزية

2026-08-15

task-review

المهن الحاسوبية الأخرى

SkillsBench task PR review — classifies the task track (standard / research / multimodal), runs static policy checks against the track-specific rubric, benchmarks the task across oracle plus Claude and Codex (with and without skills), audits trajectories for…

لغة النص الأصلي: الإنجليزية

2026-08-15

benchflow

مطوّرو البرمجيات

لغة النص الأصلي: الإنجليزية

2026-07-06

code-specialist

مطوّرو البرمجيات

Delegate complex coding tasks to a specialist model. Use when facing algorithmic challenges, performance optimization, or tricky debugging that benefits from focused code expertise.

لغة النص الأصلي: الإنجليزية

2026-06-09

mesh-analysis

تقنيو الهندسة (باستثناء رسامي المخططات)، جميع الآخرون

Analyzes 3D mesh files (STL) to calculate geometric properties (volume, components) and extract attribute data. Use this skill to process noisy 3D scan data and filter debris.

لغة النص الأصلي: الإنجليزية

2026-06-09

citation-management

معلمو التعليم العالي، جميع الآخرون

Comprehensive citation management for academic research. Search Google Scholar and PubMed for papers, extract accurate metadata, validate citations, and generate properly formatted BibTeX entries. This skill should be used when you need to find papers, verify…

لغة النص الأصلي: الإنجليزية

2026-06-09

xlsx

علماء البيانات

Comprehensive spreadsheet creation, editing, and analysis with support for formulas, formatting, data analysis, and visualization. When Claude needs to work with spreadsheets (.xlsx, .xlsm, .csv, .tsv, etc) for: (1) Creating new spreadsheets with formulas and…

لغة النص الأصلي: الإنجليزية

2026-06-09

conformance-writer

محللو ضمان جودة البرمجيات والمختبرون

Write the conformance smoke output file when the task asks for conformance.txt.

لغة النص الأصلي: الإنجليزية

2026-06-04

skill-creator

المهن الحاسوبية الأخرى

Create a tiny task-specific skill pack for BenchFlow self-gen smoke tests.

لغة النص الأصلي: الإنجليزية

2026-06-04

independent-review

محللو ضمان جودة البرمجيات والمختبرون

Incorporate feedback from an independent code reviewer to improve your solution. The reviewer is a different agent that analyzed your work.

لغة النص الأصلي: الإنجليزية

2026-06-03

code-specialist

مطوّرو البرمجيات

Delegate complex coding tasks to a specialist model. Use when facing algorithmic challenges, performance optimization, or tricky debugging that benefits from focused code expertise.

لغة النص الأصلي: الإنجليزية

2026-05-17

عرض ١٥ من أصل ١٥ skills مجمعة.

التثبيت بمساعدة مساعد ذكي

انسخ هذا الطلب إلى المساعد الذكي الذي تستخدمه.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/benchflow-ai/benchflow.