ek

How to build and extend ek's AGENT/ASSISTANT evaluation instance (ek/agents/) — the second concrete instance after ek.ocr. Covers the episode object (Episode/Trajectory/Step/Cost/RunProvenance as Layer B) and the task/tool grammar (ToolSpec/TaskSpec building a GraphGrammar as Layer A, where FieldSpec.importance is the cost of a wrong tool argument), the agent metrics (TaskSuccessMetric with an injected Checker oracle, CostPerSuccessMetric implementing Cost-of-Pass E[cost]/P(success), ToolCallMetric doing BFCL AST match over a multiset of calls, TrajectoryMetric doing a Needleman-Wunsch sequence edit distance), the reliability estimators (pass@k capability vs pass^k reliability — pure functions, NOT Metrics), the LLM-as-judge seams (JudgeSignal reference-free vs JudgeMetric reference-based vs pairwise_judge, plus judge_validation reusing Krippendorff/Cohen IAA), the agent harness (run_suite with k trials, and the variance-aware agent_regression_gate using Wilson/bootstrap intervals), and the duck-typed bridges

How to add a reference-based (offline) metric to ek's score() side — the output-type to metric to library decision table (CER/WER via jiwer, ANLS*/anls_star, span-F1 via seqeval/nervaluate, TEDS/GriTS for tables, cost-weighted typed-graph GED via networkx/apted/zss, coref via metametric), the Metric Protocol __call__(pred, gold, *, grammar=None) -> float, the Report/Score decomposition object, how to wrap a library behind the Protocol and register it, and the flagship cost-weighted typed-graph must-build whose weights come from Layer-A importance. Use when implementing, wrapping, registering, choosing, or debugging an offline/reference-based metric, scoring pred vs gold, slot/entity F1, table TEDS, graph edit distance, ANLS, WER/CER, canonicalization-before-scoring, match schemes, or per-slice/per-path TP/FP/FN. For online/reference-free confidence, calibration, and decisions use the estimate_quality side instead.

Licensing guardrail for ek developers: consult BEFORE adding ANY dependency. Covers the license-tiering policy (lean permissive core vs opt-in extras), the register of 6 license landmines (GPL Levenshtein/python-Levenshtein, LGPL TorchCP, non-commercial surya-ocr, GPL Potato, proprietary Prodigy) with their permissive replacements, scanner-invisible traps where license terms live in repo files not PyPI metadata (zss BSD-3, mistralai namespace pkg, PubTabNet CDLA), the rule that nothing copyleft or non-commercial is ever a default, and the CI license gate (pip-licenses/reuse) that fails the build on GPL/AGPL/non-commercial in core..hitl. Use when picking a metric/calibration/validation/OCR/agreement/HITL backend, adding to pyproject extras, choosing an edit-distance or conformal or annotation library, hitting a "GPL"/"non-commercial"/"RAIL"/"copyleft" license, deciding which extra a dep belongs in, or whether a dep needs import-only quarantine.

How to add a reference-free quality-estimation component to ek's online estimate_quality() side: a new Signal (deterministic verifier / intrinsic-confidence / logprob / agreement-ROVER / auxiliary-QE / LLM-judge), a Calibrator (Platt/Temperature/Isotonic), or a DecisionPolicy/SelectivePolicy (ConformalGate / RiskControlGate / CostSensitiveGate). Use when building or extending the signal->calibrate->decide pipeline, wiring a Strategy Protocol into the registry, adding confidence/uncertainty/agreement signals, ROVER voting, conformal prediction (MAPIE/crepes/TorchCP), calibration (netcal/sklearn), ECE/D-ECE, risk-coverage curves, or selective prediction / accept-flag-block gating for extraction or OCR quality. Not for offline reference-based metrics (that is the score()/add-metric side).

Master orientation map for developers BUILDING ek (a framework for building Knowledge Evaluation systems that evaluate information-extraction outputs; OCR is the noisiest special case). Read this FIRST before touching any ek component. Covers the two-need 2x2 (reference-based vs reference-free, instance vs system), the two-layer data model (Layer A GraphGrammar schema/cost SSOT + Layer B AnnotatedExtraction metadata), the facades score()/evaluate()/estimate_quality(), the strategy-Protocol + registry + dependency-injection + requires_extra open-closed pattern, the module layout (base, registry, stores, metrics, canonicalize, qe/rover, validate, harness, ocr, tools, __main__), the config2py.AppData + dol persistence model, the must-builds (two-layer object, cost-weighted typed-graph distance, ROVER), and a which-report-for-which-component table. Use when deciding where code goes, what a component is named, how facades share the Layer-A object, how strategies plug in, or which ek_0N report to read.

Build and maintain ek's OCR evaluation layer on top of the ocracy package — the concrete instance of ek's reference-based score() and reference-free estimate_quality() facades applied to image->OcrResult engines. Use when working on the ek[ocr] optional extra, wiring or running the OCR offline benchmark (CER/WER/ANLS per-slice over a gold corpus), mapping OcrResult/TextBlock into AnnotatedExtraction/FieldEstimate, deciding which ocracy backend to evaluate, handling null-safe VLM engines (Claude/GPT-4o/Mistral/Mathpix returning text-only, blocks=[], confidence=None), reading per-engine capability profiles (who emits real vs model-guessed geometry, who is calibrated, who emits TEDS-able tables), gating remote engines that make paid API calls, dealing with the missing normalized table type, or anything touching ek<->ocracy. Enforces the dependency-direction rule: ek depends on ocracy via an extra; never ocracy->ek; ek core depends only on the OcrResult shape so it evaluates any image->OcrResult callable.