smithers-eval

name	smithers-eval
description	Run a workflow over a JSON/JSONL eval suite and write a regression report. Run `bunx smithers-orchestrator eval --help` for usage details.
requires_bin	bunx
command	bunx smithers-orchestrator eval

bunx smithers-orchestrator eval

Run a workflow over a JSON/JSONL eval suite and write a regression report.

Arguments

Name	Type	Required	Description
workflow	string	yes	Path to a .tsx workflow file

Options

Flag	Type	Default	Description
--cases	string		JSON or JSONL eval case file
--suite	string		Stable suite ID used in run IDs and report paths
--runLabel	string		Run label appended to eval run IDs; defaults to current UTC timestamp plus a nonce
--dryRun	boolean	false	Plan the suite without launching runs
--concurrency	number	1	Number of eval cases to run at once
--maxCases	number		Run only the first N cases
--report	string		Write report JSON to this path
--force	boolean	false	Overwrite an existing eval report
--includeOutput	boolean	true	Include workflow outputs in the report
--maxConcurrency	number		Per-workflow max task concurrency
--root	string		Tool sandbox root directory
--log	boolean	true	Enable NDJSON event log file output
--logDir	string		NDJSON event logs directory
--allowNetwork	boolean	false	Allow bash tool network requests
--maxOutputBytes	number		Max bytes a single tool call can return
--toolTimeoutMs	number		Max wall-clock time per tool call in ms
--optimization	string		Apply a Smithers optimization artifact while running the eval suite