speq-evaluator

name	speq-evaluator
description	Use this skill when working on the speq monorepo to add or change LLM evaluation specs, extend judgement rules, modify the Bun library in packages/speq, or validate behavior with the demo app in examples/demo.

speq Evaluator

This skill is for changes inside the speq monorepo.

Use it when the task involves:

• adding or changing LLMSpec-driven evaluation behavior
• editing the Bun library in packages/speq
• changing table output or judgement rules
• updating the runnable demo in examples/demo
• verifying the workspace with the root Bun scripts

Repository Shape

• Library package: packages/speq
• Demo app: examples/demo
• Root workspace scripts: package.json
• Nix dev shell with bun and node: flake.nix

Default Workflow

1. Read README.md and the relevant package files before editing.
2. If the change affects library behavior, inspect:
   • packages/speq/src/index.ts
   • packages/speq/src/core/runner.ts
   • packages/speq/src/core/judge.ts
   • packages/speq/src/core/format.ts
3. If the change affects validation, update or add tests under packages/speq/test.
4. If the change should be demonstrated interactively, update examples/demo/src/main.ts.
5. Verify from the repo root.

Verification Commands

Run from the repository root:

bun run build
bun run test
bun run check
bun run demo

If vitest needs node from the dev shell, prefer:

nix develop -c bash -lc 'bun run check && bun run demo'

Implementation Notes

• Keep the public API centered on defineConfig, runSpeq, and formatSpeqTable.
• speq should stay provider-agnostic at the package level. Do not introduce provider-specific SDK usage into packages/speq.
• The current language judgement is heuristic and only targets Japanese and English.
• Preserve workspace boundaries:
  • reusable library code belongs in packages/speq
  • usage examples belong in examples/demo
• When adding new behavior, prefer covering it with vitest tests before relying on the demo.

Common Tasks

Add a new judgement rule

• Extend the relevant types in packages/speq/src/types.ts
• Implement the rule in packages/speq/src/core/judge.ts
• Surface the result in packages/speq/src/core/format.ts if it should appear in the table
• Add tests in packages/speq/test

Change execution behavior

• Update packages/speq/src/core/runner.ts
• Keep result ordering stable: input spec order first, model order second
• Preserve per-model error capture instead of failing the whole run

Update the demo

• Edit examples/demo/src/main.ts
• Keep the demo self-contained unless the task explicitly requires real providers or credentials

Done Criteria

Before finishing, aim to have:

• library code updated in packages/speq
• tests updated when behavior changed
• root bun run check passing
• bun run demo still producing a sensible table