eval-harness-updater

name	eval-harness-updater
description	Refresh evaluation harnesses with live/fallback parser reliability, SLO gates, and regression checks.
version	1.1.1
model	sonnet
invoked_by	both
user_invocable	true
tools	["Read","Write","Edit","Glob","Grep","Bash","Skill","MemoryRecord","WebSearch","WebFetch"]
args	--harness <path-or-name> [--trigger reflection|evolve|manual]
error_handling	graceful
streaming	supported
verified	true
lastVerifiedAt	2026-02-28
source	builtin
trust_score	100
provenance_sha	f594f3fa3d95dc6a

Eval Harness Updater

Refresh eval harnesses to keep live + fallback modes actionable under unstable environments.

Focus Areas

• Prompt and parser drift
• Timeout/partial-stream handling
• SLO and regression gates
• Dual-run fallback consistency

Workflow

1. Resolve harness path.
2. Research test/eval best practices (Exa + arXiv — see Research Gate below).
3. Add RED regressions for parsing and timeout edge cases.
4. Patch minimal harness logic.
5. Validate eval outputs and CI gates.
6. Resolve companion artifact gaps (see Cross-Reference table below).

Research Gate (Exa + arXiv — BOTH MANDATORY)

Before proposing harness changes, gather current best practices:

1. Use Exa for implementation and ecosystem patterns:
   • mcp__Exa__web_search_exa({ query: 'LLM eval harness 2025 best practices' })
   • mcp__Exa__get_code_context_exa({ query: 'eval harness parser reliability timeout handling' })
2. Search arXiv for academic research on evaluation methodology (mandatory):
   • Via Exa: mcp__Exa__web_search_exa({ query: 'site:arxiv.org LLM evaluation harness 2024 2025' })
   • Direct API: WebFetch({ url: 'https://arxiv.org/search/?query=LLM+evaluation+harness&searchtype=all&start=0' })
3. Record decisions, constraints, and non-goals in memory learnings.

arXiv is mandatory (not fallback) when topic involves: LLM evaluation, agent evaluation, SLO gates, regression testing methodology, or parser reliability.

Cross-Reference: Creator Ecosystem

This skill is part of the Creator Ecosystem. When research uncovers gaps, trigger the appropriate companion creator:

Gap Discovered	Required Artifact	Creator to Invoke	When
Domain knowledge needs a reusable skill	skill	Skill({ skill: 'skill-creator' })	Gap is a full skill domain
Existing skill has incomplete coverage	skill update	Skill({ skill: 'skill-updater' })	Close skill exists but incomplete
Capability needs a dedicated agent	agent