evals

name	evals
description	Use Mighty-native eval primitives from external repos. Covers the native eval record model, local or hosted write-back flows, and when to reach for the deeper `external-evals` skill for framework-specific mapping.

Mighty: Evals

Use this skill when a repo wants to work with Mighty-native eval records directly.

Core model

• Mighty evals are durable records, not an opaque report blob.
• The native quartet is:
  • eval_dataset
  • eval_case
  • eval_run
  • eval_result
• Keep these records authoritative even if the source runner is an external framework.
• Use file storage for large raw artifacts, traces, transcripts, or serialized source data; keep normalized fields queryable on the records.

Upstream references:

• Native Evaluation Framework
• External Python eval execution via Mighty SDK
• Pydantic Evals mapping to native eval records
• Hosted action sandboxes run repo-defined eval scripts with native write-back

Execution modes

• Local app-owned execution:
  • the repo runs its own eval logic and writes back through Mighty primitives or mighty_python_sdk
• Mighty-hosted offload:
  • an action sandbox runs the repo-defined eval script
  • the script still writes back through the same native eval path

Do not invent a separate hosted-only persistence model.

Practical rules

• Treat eval_dataset and eval_case as the durable input corpus.
• Treat eval_run as one candidate execution over a dataset or slice.
• Treat eval_result as the case-level outcome record used for regression review and comparisons.
• Preserve history. New runs append new eval_run and eval_result records instead of overwriting prior results.
• Keep framework-specific blobs in attachments or metadata, not as a replacement for normalized fields.

Repo workflow

1. Import or define the dataset and cases.
2. Start an eval_run for the candidate being measured.
3. Write one eval_result per case outcome.
4. Finish the run with summary fields that are cheap to query.
5. Attach bulky artifacts by file ref instead of embedding them inline.

When to use other skills

• Use external-evals/SKILL.md when you need deep framework mapping, Python SDK examples, or Pydantic Evals-specific write-back details.
• Use mighty/records/ when you are defining adjacent custom record types rather than working within the built-in eval quartet.

Examples

• See native-eval patterns for the native model and common repo-side write-back flows.