| name | av2-judge-pipeline |
| description | Use when implementing or reviewing the AV2 LLM-as-a-Judge execution pipeline. Do not use for generic Python setup, standalone SQL analysis, or rubric design without execution changes. |
AV2 Judge Pipeline
1. Purpose
Implement and validate the AV2 evaluation flow from candidate answers to structured judge outputs persisted in PostgreSQL.
2. When to use
Use this skill when the task involves:
- building judge prompts;
- executing judge models;
- parsing judge output;
- persisting judge evaluations;
- linking evaluations to candidate answers, rubrics, prompts, and execution metadata;
- adding or reviewing judge pipeline CLI commands.
3. When not to use
Do not use this skill for:
- initial Python project setup;
- pure database import validation;
- standalone SQL reporting;
- PostgreSQL backup/restore;
- generic code review;
- rubric authoring without execution work.
4. Required inputs
Identify:
- dataset: J1 or J2;
- question records;
- candidate answers from AV1;
- reference answer, answer key, or rubric;
- judge model identifier;
- prompt version;
- rubric version;
- expected output schema;
- database connection method;
- execution mode and parameters.
5. Required commands
Use project-specific commands when available.
Baseline command pattern:
.venv/bin/python -m atividade_2.cli --help
.venv/bin/python -m atividade_2.cli run-judge --help
.venv/bin/python -m pytest
Inspect changed files:
git diff --stat
git diff -- src tests
6. Workflow
Step 1 — Read project context
Load PRIMING.md.
Confirm:
- dataset semantics;
- J1/J2 evaluation rules;
- judge output contract;
- chain-of-thought handling rule;
- database traceability expectations.
Step 2 — Map the data flow
Document:
question
+ candidate answer
+ reference/rubric
+ judge prompt version
-> judge request
-> raw judge response
-> parsed structured output
-> validated score/rationale
-> database row
Step 3 — Build prompts from explicit inputs
The judge prompt must include:
- original question;
- candidate answer;
- reference, answer key, or rubric;
- scoring scale;
- hallucination penalty instruction;
- verbosity neutrality instruction;
- machine-parseable output schema.
Step 4 — Validate judge output
Require:
- valid JSON or explicitly parseable structure;
- integer score from 1 to 5;
- non-empty structured rationale;
- hallucination risk field;
- rubric alignment field;
- human review flag.
Reject or quarantine malformed outputs. Do not silently coerce invalid records.
Step 5 — Persist traceable evaluation
Every persisted evaluation must link to:
- question;
- candidate answer;
- candidate model;
- judge model;
- prompt version;
- rubric version;
- score;
- rationale;
- timestamp;
- execution metadata.
Step 6 — Test before broad execution
Add tests for:
- prompt builder;
- parser;
- invalid output handling;
- score range enforcement;
- persistence mapping;
- idempotency or duplicate behavior.
Run a small fixture before full dataset execution.
7. Output format
## Context & Goal
## Data flow
## Inputs verified
## Implementation summary
## Judge output contract
## Persistence mapping
## Validation performed
## Risks and follow-ups
8. Validation checklist
9. Guardrails
- Do not evaluate answers without reference/rubric context.
- Do not compare candidate answers against each other as gold.
- Do not reward verbosity by itself.
- Do not persist hidden reasoning.
- Do not silently accept invalid JSON or invalid scores.
- Do not overwrite existing evaluations unless explicitly requested and versioned.
