pptx-oracle-eval

name	pptx-oracle-eval
description	Use when running, refreshing, comparing, or summarizing PPTX renderer oracle evaluations, including shape, SmartArt, chart, table, connector, fillstroke, python-pptx, cases, windows-cases, and regression report workflows in this repository.

PPTX Oracle Eval

Overview

Run PPTX renderer oracle evaluations without mixing stale reports, loose pytest thresholds, and high-fidelity support gates.

Before Running

Capture provenance first:

pwd
git status --short

Ensure E2E servers are available. If needed, start pnpm dev:e2e in a separate long-running session, then verify:

curl -sS http://127.0.0.1:8080/api/testdata-files >/dev/null
curl -sS http://127.0.0.1:5173/test/pages/e2e-compare.html >/dev/null

If ownership is unclear, reuse healthy servers.

Pick The Corpus

User intent	Command shape
One shape ID	--shape-id-min N --shape-id-max N
Shape range	--shape-id-min A --shape-id-max B
SmartArt only	--smartart-cases-dir oracle/cases-full
Full curated oracle cases	--cases-dir oracle/cases-full
Python-pptx cases	--pypptx-cases-dir oracle/cases-pypptx
Windows ground truth	add --source windows

Use semantic output names:

cd test/e2e
.venv/bin/python3 scripts/run_all_shapes_eval.py \
  --cases-dir oracle/cases-full \
  --out reports/oracle-failures/<topic>-$(date +%Y%m%d-%H%M%S).json

For one known case, use the API first:

cd test/e2e
curl -sS -X POST http://127.0.0.1:8080/api/evaluate/CASE \
  > reports/oracle-failures/CASE-probe-$(date +%Y%m%d-%H%M%S).json

Quality Gate

Use the repository's two-layer metric model:

Layer	Rule
Automated pass/fail	ssim >= 0.95 and color_hist_corr >= 0.80
Human review	ssim < 0.99 sets needs_review=true
Diagnostic only	fg_iou, fg_iou_tolerant, chamfer_score, mae

Do not change thresholds during an eval run.

Do not update baselines, manual verdicts, or support-catalog.json unless asked.

Summarize The Run

Report counts and worst cases:

jq '{generated_at,total_cases,
  passed:(.results|map(select(.passed==true))|length),
  failed:(.results|map(select(.passed==false))|length),
  needs_review:(.results|map(select(.needs_review==true))|length),
  errors:(.errors|length)}' <report>.json

jq -r '.results
  | sort_by(.summary.ssim)
  | .[:20][]
  | [.case,.summary.ssim,.summary.color_hist_corr,.summary.fg_iou_tolerant,.summary.chamfer_score,.passed,.needs_review,((.reasons // [])|join(";"))]
  | @tsv' <report>.json

For mixed corpora, group results by domain (chart, smartart, shape, table, connector, fillstroke, text, composite) before recommending next work.

Interpret Carefully

• Treat generated_at and mtime as evidence.
• Separate current facts from older report history.
• If metrics conflict with the UI, re-run the single case via POST /api/evaluate/{case}.
• Chart-heavy failures need XML inspection, not blind ECharts tuning.
• SmartArt near misses need manual review even when gates pass.

Red Flags

• Reporting default reports as current without checking generated_at.
• Starting a foreground dev server and never running eval.
• Running full corpus when the user named one case.
• Updating baselines/support status as measurement side effect.

Reporting Back

Include command, report path, generated_at, corpus/source, git dirty status, pass/fail/review/error counts, top cases by domain, stale data, and next target.