external-evals

name	external-evals
description	Use Mighty native eval records from external repos, including local Python eval scripts and optional Mighty-hosted action execution that writes back through the same SDK contract.

External Evals

Use this skill when an external repo needs to store evaluation datasets, cases, runs, and results in Mighty without moving evaluation ownership into Mighty.

Core model

• Mighty stores eval state in four native record types: eval_dataset, eval_case, eval_run, and eval_result.
• Keep the native quartet authoritative even when the source framework is pydantic_evals, PydanticAI, or another external runner.
• Prefer compact, queryable normalized fields inline. Put large raw payloads, serialized source datasets, full report JSON, traces, and transcripts in Mighty file storage and reference them with mighty-file:<file_id>.

Upstream references:

• Native Evaluation Framework
• External Python eval execution via Mighty SDK
• Pydantic Evals mapping to native eval records
• Hosted action sandboxes run repo-defined eval scripts with native write-back

Execution modes

• Local app-owned execution: the repo runs its own Python script, agent, dependencies, concurrency, and retries, then calls mighty_python_sdk.
• Mighty-hosted offload: an action_def runs that same repo-defined script inside a sandbox. The script still calls the Mighty SDK to persist eval_run and eval_result records. Do not invent a hosted-only write path.

Python pattern

Use mighty_python_sdk.MightyClient(...).evals or AsyncMightyClient(...).evals.

Typical flow:

from mighty_python_sdk import MightyClient

with MightyClient() as client:
    source_upload = client.evals.upload_attachment_path("evals/support_cases.json")

    imported = client.evals.import_dataset(
        name="support-benchmark",
        version="2026-03-12",
        split="dev",
        source="pydantic_evals",
        source_file_ref=source_upload.ref,
        metadata={"framework": "pydantic_evals"},
        cases=[
            {
                "case_identifier": "refund-policy",
                "name": "refund-policy",
                "input": {"question": "What is the refund window?"},
                "expected": {"answer": "30 days"},
                "metadata": {"tier": "gold"},
            }
        ],
    )

    run = client.evals.start_run(
        name="support-benchmark / candidate-a",
        dataset_id=imported.dataset.id,
        candidate_name="candidate-a",
        candidate_ref="git:abc123",
        framework="pydantic_evals",
    )

    client.evals.write_result(
        eval_run_id=run.id,
        eval_case_id=imported.cases_by_identifier["refund-policy"].id,
        name="support-benchmark / refund-policy",
        status="passed",
        output={"answer": "30 days"},
        scores={"exact_match": 1.0},
        assertions={"exact_match": True},
        metrics={"latency_ms": 412},
    )

    client.evals.finish_run(
        run.id,
        status="completed",
        summary={"case_count": 1, "failure_count": 0},
    )

Pydantic Evals mapping

• Map one Dataset to one eval_dataset.
• Map each Case to one eval_case using a stable case_identifier. Prefer the source case name when it is stable.
• Map one evaluation execution to one eval_run.
• Map each ReportCase or ReportCaseFailure to one eval_result.
• Preserve scores, labels, assertions, metrics, attributes, trace_id, span_id, and source_case_name when present.
• If the report contains repeated executions of the same source case, do not write them into one eval_run if that would duplicate eval_run_id + eval_case_id. Split repeats into separate runs or aggregate before write-back.

SDK conveniences:

• client.evals.import_pydantic_dataset(...)
• client.evals.write_pydantic_report_results(...)
• client.evals.finish_pydantic_run(...)

Review and query conventions

• Compare candidates at the eval_run level; review failures and regressions at the eval_result level.
• Put raw serialized datasets and reports in file storage, then attach the refs on the dataset or run.
• Keep framework-specific blobs as attachments or nested metadata. Do not replace native normalized fields with one opaque framework blob.

Hosted action guidance

• If Mighty is offloading execution, define an action that runs the repo’s script or module directly.
• Pass auth and environment through the action sandbox, but let the script own evaluation logic.
• Preserve action_run_id on the resulting eval_run so hosted executions stay traceable.