terminal-agent-benchmark

Run and maintain Con's terminal-agent benchmark against a live app session. Use when validating con-cli, SSH workspace reuse, tmux awareness, agent-target preparation, or when collecting benchmark evidence for regressions and release notes.

Jump to install

Repository: nowledge-co/con-terminal
Last source activity: April 30, 2026 at 15:25
Detected SKILL.md language: English
Stars: 556
Forks: 44

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/nowledge-co/con-terminal --skill terminal-agent-benchmark

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

SKILL.md

Source instructions · Read-only preview

Back to search Back to installation

release

Run and verify Con beta, dev, and stable releases without publishing incomplete artifacts.

2026-08-27

con-cli-e2e

Validate Con's local socket control plane against a real running app session, and write/run con-test integration tests. Use when testing con-cli, the Unix socket API, pane control, tmux control, in-session agent calls, or when writing E2E test cases in crates/con-test/testdata/.

2026-05-08

changelog-release-notes

Maintain Con's CHANGELOG.md and release notes. Use when updating changelog entries, preparing a beta/dev release, reviewing PR release-note coverage, or ensuring contributor PR credit is present.

2026-05-07

View all from this repository

Terminal Agent Benchmark

Use this skill when you need to evaluate Con as a terminal-native agent, not just compile it.

Primary references:

Default workflow

Confirm a live app session and socket exist.

Run the strict benchmark:

python3 benchmarks/terminal-agent/run.py --suite strict

If provider setup is present, run:

CON_BENCH_ENABLE_AGENT=1 python3 benchmarks/terminal-agent/run.py --suite all

Prefer a built-in profile when one matches the workflow:

python3 benchmarks/terminal-agent/run.py --list-profiles
python3 benchmarks/terminal-agent/run.py --profile basic-local-shell
CON_BENCH_ENABLE_AGENT=1 python3 benchmarks/terminal-agent/run.py --profile basic-local-codex --suite all

Use starter profiles for quick regression checks and operator profiles for richer coding, SSH maintenance, or tmux dev-loop evaluation.

python3 benchmarks/terminal-agent/run.py --profile operator-local-codex-devloop --suite operator
python3 benchmarks/terminal-agent/run.py --profile operator-local-claude-devloop --suite operator
python3 benchmarks/terminal-agent/run.py --profile operator-local-opencode-devloop --suite operator
python3 benchmarks/terminal-agent/run.py --profile operator-ssh-dual-host-maintenance --suite operator
python3 benchmarks/terminal-agent/run.py --profile operator-ssh-tmux-devloop --suite operator

For SSH/tmux changes, run the relevant playbook under benchmarks/terminal-agent/playbooks/.

Save the JSON record under .context/benchmarks/ and cite it in your report.

If the run is an operator benchmark, score it with:

python3 benchmarks/terminal-agent/score.py --profile ... --record ... --score ...

Generate a trend report when comparing many runs:

python3 benchmarks/terminal-agent/report.py

Rules

Prefer pane_id over pane_index when following up on benchmark findings.

Keep benchmark control on the existing panes.* API unless the benchmark explicitly targets pane-local surfaces. Surface support is additive and should not change the built-in agent's pane/tool contract.

Do not treat playbook observations as strict pass/fail evidence unless the behavior is actually deterministic.

If a scenario depends on host setup, say so explicitly.

Keep operator playbooks safe-by-default. Prefer read-only checks first, and treat destructive or privileged steps as explicit branches.

If a benchmark reveals a product limit, document the limit instead of hiding it behind a softer assertion.

Operator suites intentionally serialize agent ask turns. If a tab already has a pending agent request, let the runner wait and reuse the same tab instead of opening a parallel benchmark against it.