| name | excel-extraction-pipeline-improver |
| description | Iteratively improves the xlsx_parser extraction pipeline using feedback, test failures, and golden outputs. Use when fixing extraction gaps, addressing parser feedback, or improving coverage on stress workbooks. Follows TDD with instrumentation. |
Feedback-Driven Extraction Pipeline Iteration
Given extraction output, test failures, and user feedback, iteratively improve the Excel ingestion pipeline until it passes golden tests and reaches high coverage.
Inputs
| Input | Description |
|---|
repo_path | Path to the xlsx_parser repository |
stress_workbooks_dir | Directory containing stress test workbooks |
golden_expected_dir | Golden expectations (expected_extraction_min.json) |
current_output_dir | Latest extraction output |
feedback | Human notes, failure logs, diffs |
time_budget_minutes | Default 120 |
Operating Procedure (TDD + Instrumentation)
1. Baseline
- Run unit and integration tests
- Run extraction on stress workbooks
- Produce coverage report with:
- % cells with value
- % cells with formula
- % tables detected
- % charts detected
- % objects with anchors/bboxes
- % conditional formats captured
2. Triage
Classify each failure:
| Class | Meaning |
|---|
| Parsing gap | Not extracted |
| Schema gap | Extracted but not represented in DTO |
| Normalization gap | Extracted but wrong type / wrong display_text |
| Linking gap | Refs not resolved |
| Performance regression | Slower than before |
3. Fix One Class at a Time
- Add/adjust DTO fields only when needed
- Add a focused test (golden assertion) before writing code
- Implement extraction improvements
- Add debug logs (behind a flag) for formula parsing, table mapping, object anchors/bboxes
4. Regression
- Re-run extraction on all stress workbooks
- Update coverage report
- Update golden outputs only when:
- Change is clearly correct
- Accompanied by rationale in
CHANGELOG.md or test comment
Must-Have Behaviors
- Lossless references: raw formula string + resolved dependency list (ranges/names)
- Display semantics: display_text + number_format
- Layout semantics: merged cells, row/col sizing, anchors/bboxes for objects
- Deterministic output ordering for stable diffs
Deliverables per Iteration
A PR-style commit (or patch) with:
- Tests added
- Code changes
- Updated coverage report
- Before/after diff summary
Acceptance Criteria
- Passes all golden tests for
expected_extraction_min.json
- No new regressions on prior workbooks
- Coverage increases monotonically for targeted categories
- Extraction output stable across runs
Reference
See DESIGN.md for pipeline stages and src/xlsx_parser/pipeline.py for orchestration.
