القائمة
Benchmark runner, fixture repos, and publication. Read before running, modifying, or publishing benchmark results.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | benchmark |
| description | Benchmark runner, fixture repos, and publication. Read before running, modifying, or publishing benchmark results. |
ttsc + @ttsc/lint + ttsc format versus the legacy tsc + eslint + prettier toolchain, on seven real-world TypeScript projects. experimental/benchmark/performance.mjs clones each fixture's three branches into experimental/benchmark/.work/, replays each cell RUNS times, and writes the medians to website/public/benchmark/performance.json for the public dashboard at https://ttsc.dev/benchmark.
experimental/benchmark/graph.mjs is a separate AI-token benchmark for @ttsc/graph and graph-MCP comparators. It owns its fixtures: the graph branch of each benchmark repo, cloned into ../graph-benchmark-work/ beside this repo and installed from the fixture's own lockfile. It runs projects sequentially, fixes reasoning effort to high, and upserts its own cells into website/public/benchmark/graph.json. The codegraph arm runs codegraph init before the agent cells, records that setup as toolSetupMs, local-ignores .codegraph/, and removes the index unless --keep-codegraph-index is passed. The codebase-memory arm runs codebase-memory-mcp cli index_repository with an isolated CBM_CACHE_DIR, records that setup as toolSetupMs, local-ignores .codebase-memory/, and removes the cache unless --keep-codebase-memory-index is passed. The serena arm starts Serena's stdio MCP server through uvx by default, local-ignores .serena/, and removes the project metadata unless --keep-serena-project is passed.
Cell ID = project:branch:op:threading.
| Axis | Values |
|---|---|
| Project | vue, rxjs, typeorm, zod, nestjs, vscode, shopping-backend |
| Branch | legacy, ttsc, ttsc-lint |
| Op | build, noEmit, eslint (legacy only), format |
| Threading | single, checkers2, checkers4, checkers8 (format keeps single + default multi) |
Methodology, per-axis interpretation, and dashboard tabs are documented in website/src/content/docs/benchmark/performance.mdx. Flags and env vars are tabled in experimental/benchmark/README.md. This skill covers the higher-level rules that those references assume.
node experimental/benchmark/performance.mjs # full matrix sweep
node experimental/benchmark/performance.mjs --project=vue --no-website # one fixture, do not touch dashboard
node experimental/benchmark/performance.mjs --verify-only # one pass per cell, no timing
node experimental/benchmark/performance.mjs --list # print resolved grid
node experimental/benchmark/performance.mjs --sequential # clone, measure, delete one fixture at a time (low-disk)
node experimental/benchmark/graph.mjs --project=typeorm --models=gpt-5.4-mini --tools=ttsc-graph,codegraph,codebase-memory,serena # one graph AI-token fixture
node experimental/benchmark/graph.mjs --all --models=gpt-5.4-mini --arm=baseline --tools=baseline --prompt-family=all --runs=5 --reset # baseline-only publication refresh
node experimental/benchmark/graph/audit-codex-traces.mjs --dir=experimental/benchmark/.work/graph/<timestamp> # inspect Codex messages/tools/reasoning tokens and baseline savings
node experimental/benchmark/graph/audit-codex-traces.mjs --compare=<before>,<after> # compare smoke-run audit/report dirs during optimization
node experimental/benchmark/graph/audit-codex-traces.mjs --self-test # regression-check audit parsing and savings semantics
Option families:
--project, --cell-filter, --lint-only, --format-only, --ttsc-build-only. Every targeted run pairs with --no-website so the dashboard stays consistent.--setup-only, --no-setup, --no-install, --no-pack (or TTSC_BENCH_SKIP_PACK=1), --force-install. Setup packs the local ttsc workspace into tarballs, clones the fixtures, installs the tarballs, and runs ttsc prepare so plugin binaries are warm before any measured cell.TTSC_BENCH_RUNS (5), TTSC_BENCH_WARMUP (1), TTSC_BENCH_RETRIES (2, applied only to race-classified failures). The reported number is the median; min and the full sample list stay in report.json for audit.TTSC_BENCH_REQUIRE_QUIET=1 upgrades the load-average warning into a hard error and is set for every publication run; TTSC_BENCH_SKIP_LOAD_CHECK=1 silences it for development iterations.--no-website skips merging into website/public/benchmark/performance.json; --reset discards prior measurements instead of merging in place; TTSC_BENCH_OUT redirects the local report; --verbose tees child stdio with [cmd] / [step] / [timer] traces (default is milestone-only).graph.mjs --arm=baseline --tools=baseline records only the empty-MCP baseline. Later --arm=graph --tools=ttsc-graph,codegraph,codebase-memory,serena runs add comparator samples against those website baseline cells. Prompt families are dedicated and common; project-specific and shared-onboarding remain accepted aliases. Graph-arm fallback to shell source reads/searches is a real measured outcome, not an invalid result; keep it visible through shell, sourceTouches, graph, and attempts. Exclude only zero-token infrastructure/capacity failures from published medians.--no-website and unique --out directories, then publish afterward with node experimental/benchmark/graph/publish.mjs --from <out-dir> for each completed suite. Direct concurrent writes to website/public/benchmark/graph.json can race.graph branch, outside the repo: graph.mjs clones each benchmark repo's graph branch into ../graph-benchmark-work/<project>@graph, installs it from its own lockfile, and never touches the performance sweep's .work/ clones. The branch split keeps a graph-only fixture edit (a tsconfig whose program includes the tests) from changing what the tsc-vs-ttsc cells compile. The directory split keeps ttsc's own CLAUDE.md / AGENTS.md out of the measurement: the agent's cwd is the fixture, and both CLIs walk the parent chain for them (a vscode run under experimental/benchmark/.work was caught reading this repo's AGENTS.md instead of touring vscode). A graph number measured from a fixture under this repo is contaminated — re-measure it, do not publish it.vue@graph), never ttsc-benchmark-vue@… — the prefix makes an agent hunt for harness code instead of touring the source.vscode graph benchmark runs are globally single-lane. Do not run two vscode graph cells at the same time, even if the tool, model, or prompt family differs. Other projects may run in parallel while one vscode cell runs.runs: 1 in the website JSON. (The performance.mjs N=5 median rule for the tsc-vs-ttsc dashboard is separate and unchanged.)common and dedicated for every fixture and diffed cell-by-cell against the currently published graph.json. A cell that loses reduction, gains tool calls, or newly reads files is a regression that blocks merge until its trace explains the cause and the cause is fixed. Because the runs are N=1, you cannot average a bad cell away or wave it off as "inherent" (a prior model answered the same prompt in one call, so "inherent" is almost always wrong); one non-ideal cell means the change is unfinished, and it is cleared only by reading the trace, never by the token percentage. Validating one family, or the average, and declaring victory is how the last dedicated regression shipped unseen.--max-run-retries=4 so transient agent failures do not thin the median. N=1 optimization smoke runs may use --max-run-retries=0 when the point is to expose a failure mode without spending tokens on repeated attempts.graph.mjs automatically writes codex-trace-audit.json beside the suite report whenever it runs Codex cells; run audit-codex-traces.mjs directly only to re-audit an existing run. The audit records every exposed assistant message, every shell/MCP call in timeline order, per-turn usage, and reasoning_output_tokens; hidden reasoning text is not emitted by Codex and must not be invented. It separates strict exact avoidable output such as duplicate MCP calls and legacy inline evidence text, measured graph-replaceable shell-output surface, candidate MCP overfetch surfaces such as broad graph traces, later-turn prompt replay exposure where Codex exposes multiple turn.completed events, zero-MCP or shell-fallback graph-arm traces, and an input ledger that compares usage input tokens with visible trace material. The ledger's unexplained input is an accounting gap, not proof of one hidden category. It reports observed, replacement lower-bound, candidate-ceiling, and observed replay-adjusted savings against the matching N=5 website baseline medians. Use --compare=<before>,<after> on audit JSON files, suite reports, or suite directories to compare optimization smoke runs by the same token/reasoning/tool/savings fields. Pass --baseline=none to disable baseline comparison.<!-- ttsc-benchmark-results -->, include the latest table, the report/audit paths, and known invalid/missing cells, and update that same comment whenever a newer measurement supersedes it. If no PR exists yet, keep the table in the final report and mark the PR comment as pending; post/update it immediately after the PR exists.--sequential (env TTSC_BENCH_SEQUENTIAL=1) holds only one (project, branch) clone in .work/ at a time: clone, measure, delete, next. Incompatible with --setup-only / --no-setup.website/public/benchmark/performance.json. merge-website.mjs remains available when merging partial report.json files by cell id.Publication sweep:
TTSC_BENCH_REQUIRE_QUIET=1 node experimental/benchmark/performance.mjs
After the sweep, inspect the diff against website/public/benchmark/performance.json: every fixture row present, row order preserved, host panel reflects the machine that produced the numbers.
Each fixture is a forked GitHub repo at samchon/ttsc-benchmark-<name>, plus samchon/shopping-backend for the plugin-heavy case. Every fixture carries four independent branches:
legacy: upstream source with stock tsc, eslint, and prettier. TypeScript pinned to the Legacy TypeScript version shown on the dashboard host panel (currently v6.0.3).ttsc: same source as legacy, with tsc swapped for ttsc on the pinned TypeScript-Go typescript@rc runtime and the workspace configured to install ttsc from the tarball the runner packs.ttsc-lint: same source as ttsc, with @ttsc/lint folded into the compile pass so the eslint step is no longer invoked.graph: the branch the AI-token benchmark measures, branched from ttsc. It carries what the graph agent needs and the performance sweep must not see — above all a tsconfig.graph.json whose program includes the tests. Graph-only fixture edits go here, never to ttsc.The three branches differ only in tooling, package.json, tsconfig*.json, eslint.config.*, .prettierrc*, lockfile, ttsc plugin descriptors. Application source is identical across legacy, ttsc, and ttsc-lint, so a cell delta reflects tool cost rather than workload drift.
Across legacy and ttsc-lint, the lint and format cells process exactly the file set tsconfig.json compiles for that fixture. eslint and @ttsc/lint lint that set; prettier --check and ttsc format format that set.
Nothing inside the program is excluded; nothing outside it is targeted. Carve-outs via --ignore-pattern, eslint.config ignores, .prettierignore, or extra files/exclude entries are rejected because they make the cells incomparable.
On the graph branch, each fixture's graph program is the sources and their tests — the program an editor's language server holds open, not the emit-only build program. The benchmark question asks which tests to read next, so a graph program without tests forces the agent to glob the repo for spec files and the token win collapses (vue: 31% with a test-less program, 82% and a single tour with the tests in it). Never point a graph cell at a build config that excludes tests, and never let a fixture's graph program drift back to tsconfig.build.json.
Edits go to the fixture repo on GitHub, not to the local clone. Setup runs fetch + reset --hard to the upstream branch tip on every run, so local changes in .work/ disappear immediately.
Finish every fixture-branch edit by running the branch's own build, format, and lint commands (e.g. pnpm build, prettier --write or ttsc format, eslint --fix) until the tree is green, then commit and push. A half-finished tip pollutes every later run because the runner pulls upstream every setup.
READMEs and prose docs inside a fixture repo follow the same writing rules as ttsc itself. See AGENTS.md ## Maintenance 짠 Writing style and .codex/skills/documentation/SKILL.md 짠 READMEs.
ttsc workspace during setup. The fixture repo must not carry pre-baked tarball paths, vendored ttsc builds, or stale dist/ output.legacy branch in the same release so the tsc baseline stays one major across the matrix.samchon/ttsc-benchmark-<name> repo carrying all three branches, plus a PACKAGE_CONFIGS entry at the top of performance.mjs. Multi-fixture tricks inside one repo are out of scope.type-fest fixture was dropped deliberately. The tsgo rows (raw TypeScript-Go measured on the same ttsc clone for build/noEmit, cell ID project:ttsc:tsgo:op:threading) are a kept reference for launcher overhead, not eligible for the headline winner.Pull request submission flow. Read only when the user explicitly asks for a pull request.
Work rules, testing, validation, and change integrity. Read before writing or modifying code.
Self-Review, Review Cycle, Discussion, and Research Review Round workflows. Read the Exhaustive rounds and Stop condition rules before any review round (solo or team); read the Briefing subagents rule before delegating to any subagent; read in full when the user asks for a named mode.
What ttsc is, the workspace layout, and the canonical commands.
READMEs and website guides. Read before writing or modifying docs.
Keeping packages/ttsc/shim/* synced with typescript-go and complete for plugin authors. Read before adding a re-export, bumping the pinned typescript-go version, or chasing a missing AST/transform/printer/emit API a plugin needs.