repair-rag-artifact-contract

name

repair-rag-artifact-contract

description

Diagnose and repair RAG artifact metadata contract failures before patching downstream evaluators. Use when retrieval/eval artifacts have missing or inconsistent item metadata such as `canonical_doc_key`, `child_id`, `chunk_index`, `source_space`, or `parent_id`, or when downstream symptoms appear such as `alignment_row_count=0`, `CC_adj_lexical=0`, `SR_fragmentation_proxy=0`, empty `reranked_contexts_items` fields, or report-facing key drift.

Repair Rag Artifact Contract

Overview

Use this skill to decide whether a RAG evaluation failure is a downstream metric bug or an upstream artifact contract bug. Start from the artifact, inspect item metadata coverage, trace the producer that dropped metadata, and only then patch evaluators or reports.

Workflow

1. Confirm the symptom.
2. Inspect artifact item metadata coverage.
3. Trace the upstream producer that should have materialized the missing fields.
4. Patch producer or backfill stage first.
5. Rebuild artifacts.
6. Re-run downstream metrics and reports.

Do not start by patching eval_cc.py, eval_sr.py, or report formatting unless the artifact contract is already intact.

Quick Start

Run the contract checker on the suspect artifact first.

/Users/jaehyuntak/Desktop/Project_____현재_진행중인/my-second-identity/.venv/bin/python \
skills/repair-rag-artifact-contract/scripts/check_artifact_contract.py \
plans/codex/test_result/hete_parent_cluster_full_2026-03-15.json \
--mode hete_parent_cluster

If coverage for canonical_doc_key, child_id, source_space, or chunk_index collapses in reranked_contexts_items, treat the producer path as broken.

Symptom Routing

Use this routing before editing code.

• alignment_row_count=0
  • Check artifact metadata before touching evidence alignment logic.
• CC_adj_lexical=0 or SR_fragmentation_proxy=0
  • Check whether parent anchors and chunk order fields survived artifact creation.
• IDs remained None or reranked_contexts_items[*] fields are empty
  • Patch artifact producer or backfill stage before changing evaluators.
• report-facing key drift
  • If artifact contract is healthy, then check .claude/skills/rag-bench/knowledge_bases/metric_formula_contract.md and downstream report code.

Project Hotspots

Read references/upstream-downstream-map.md for the file map.

In this repository, the common upstream/downstream split is:

• Upstream producers / backfill
  • plans/codex/scripts/eval_ragas_llmjudge_10modes_kgrid.py
  • plans/codex/scripts/eval_rerank5_artifacts.py
  • plans/codex/algorithms/backfill_artifacts_doc_key.py
• Downstream evaluators
  • .claude/skills/rag-bench/scripts/eval_cc.py
  • .claude/skills/rag-bench/scripts/eval_sr.py
  • plans/codex/scripts/build_hete_evidence_alignment_2026-03-16.py

If upstream fields are missing, downstream metrics are secondary symptoms.

Artifact Contract Checklist

For reranked_contexts_items, check these fields first:

• text
• canonical_doc_key
• child_id
• doc_id
• parent_id
• source_space
• chunk_index

Treat canonical_doc_key as the shared document anchor unless a more specific project contract overrides it.

Rules

Use the following rules consistently.

1. Inspect artifacts before patching evaluators.
2. Prefer producer fixes over evaluator-specific workarounds.
3. Rebuild artifacts before trusting new metric outputs.
4. Keep strict metrics and executable proxy metrics distinct.
5. Use report-facing names from .claude/skills/rag-bench/knowledge_bases/metric_formula_contract.md.

Resources

• scripts/check_artifact_contract.py
  • Summarize per-field non-empty coverage for item arrays such as reranked_contexts_items.
• references/upstream-downstream-map.md
  • Map common symptoms to the upstream producer or downstream evaluator most likely involved.