pdf-markdown-validator

# PDF → Markdown Validator Skill Validate PDF to Markdown conversion quality using a comprehensive, multi-dimensional evaluation framework. This skill provides standardized metrics, evaluation harnesses, and reporting tools to assess conversion fidelity across table accuracy, style preservation, robustness, and performance. ## When to use Use this skill when you need to: - **Validate PDF extraction quality**: Measure how accurately PDFs convert to Markdown - **Compare conversion approaches**: Benchmark different PDF processing implementations - **Track quality improvements**: Quantify gains from processing enhancements - **Automate quality gates**: Enforce minimum quality standards in CI/CD pipelines - **Generate evaluation reports**: Create detailed analysis of conversion successes and failures - **Identify failure patterns**: Discover systematic issues in specific PDF features - **Measure performance regressions**: Track processing speed alongside quality metrics ## Core concepts ### Validation Framework The validation framework computes a **composite quality score (0–100)** combining four independent metric dimensions: ``` FinalScore = (0.40 × TableAccuracy) + (0.40 × StyleAccuracy) + (0.10 × Robustness) + (0.10 × Performance) ``` Each dimension is independent and can be evaluated separately or together. ### 1. Table Accuracy (40% weight) Measures how accurately tables are detected and their cell content extracted. **Components:** - **Table Detection F1**: IoU-based matching of predicted vs. gold tables (IoU ≥ 0.5 threshold) - Precision: correctly identified tables / all detected tables - Recall: correctly identified tables / all gold tables - F1: harmonic mean of precision and recall - **Cell Content Accuracy**: Token-level F1 averaging across matched table cells - Matches cells by position within detected tables - Unmatched cells score 0.0 - Aggregates to mean F1 across all cells **Formula:** ``` TableAccuracy = (0.5 × TableDetectionF1) + (0.5 × CellContentAccuracy) ``` **Interpretation:** - **90–100**: Excellent table handling; tables detected and content extracted accurately - **70–89**: Good; minor detection misses or cell content variations - **50–69**: Fair; tables detected but cells have significant content errors - **Below 50**: Poor; systematic table detection failures or content corruption ### 2. Style Accuracy (40% weight) Measures how accurately text formatting (bold, italic, heading levels) is preserved. **Components:** - **Bold F1**: Token-level F1 for bold detection - **Italic F1**: Token-level F1 for italic detection - **Heading F1**: Heading-level accuracy (prediction must match gold level exactly; e.g., H2 ≠ H3) **Formula:** ``` StyleAccuracy = macro_average(BoldF1, ItalicF1, HeadingF1) = (BoldF1 + ItalicF1 + HeadingF1) / 3 ``` **Interpretation:** - **90–100**: Styles consistently preserved; formatting markup matches gold - **70–89**: Good; minor style detection misses - **50–69**: Fair; mixed style accuracy; some styles missed consistently - **Below 50**: Poor; styles largely lost or misdocumented ### 3. Robustness (10% weight) Measures system stability and validity across a test corpus, particularly edge cases. **Components:** - **Crash-free rate**: Percent of documents processed without panics or non-zero exit codes - **Markdown validity**: Percent of generated Markdown files passing `pandoc` syntax validation - **Completeness**: Percent of documents producing non-empty output **Formula:** ``` Robustness = (CrashFreeRate + MarkdownValidityRate + CompletenessRate) / 3 ``` **Interpretation:** - **95–100**: Production-ready; system handles edge cases gracefully - **85–94**: Stable; occasional edge-case issues but recovers - **70–84**: Moderate stability; known failure patterns on specific input types - **Below 70**: Fragile; systematic crashes or invalid output on subset of inputs ### 4. Performance (10% weight) Measures processing speed relative to a baseline; targets 1-page PDF ≈ 200–500ms. **Components:** - **Median latency**: P50 processing time across corpus - **P95 latency**: 95th percentile processing time (captures tail behavior) **Formula:** ``` Performance = 0.5 × min(1.0, baseline_median / run_median) + 0.5 × min(1.0, baseline_p95 / run_p95) ``` **Interpretation:** - **95–100**: Fast processing; meets or exceeds baseline - **80–94**: Good; acceptable slowdown (≤20%) - **60–79**: Acceptable; noticeable overhead (20–60%) - **Below 60**: Slow; significant performance regression ## Quick start ### 1. Prepare ground-truth annotations Create `.gold.md` files for each test PDF: ```bash # Copy reference Markdown next to PDF with .gold.md extension cp reference_output.md test.gold.md # Format: Each section annotated with metadata # Gold format example: # # Heading 1 # **bold text** and *italic text* # # | Column A | Column B | # |----------|---------| # | cell 1 | cell 2 | ``` ### 2. Run validation ```bash # Evaluate against ground truth cargo run -p edgequake-pdf --example real_dataset_eval -- \ --input crates/edgequake-pdf/test-data/real_dataset \ --gold \ --metrics # Generate detailed report python3 .github/skills/pdf-markdown-validator/scripts/validate.py \ --pdf-dir crates/edgequake-pdf/test-data/real_dataset \ --gold-dir . \ --output-report metrics_report.json ``` ### 3. Interpret results ```bash # View summary scores cat metrics_report.json | jq '.summary' # Analyze failures by category python3 .github/skills/pdf-markdown-validator/scripts/analyze_failures.py \ metrics_report.json ``` ## Capabilities ### Core Capabilities #### 1. Metric Computation - **Token-level F1 calculation**: Boolean/classification metrics at granular text unit level - **Spatial matching**: IoU-based table detection using bounding box coordinates - **Aggregate averaging**: Macro-averaging (label-balanced) and weighted averaging strategies - **Normalization**: Robust handling of missing annotations and edge cases #### 2. Evaluation Harness - **Batch evaluation**: Process entire corpora in single invocation - **Incremental scoring**: Support partial annotations and progressive refinement - **Performance profiling**: Automatically capture timing with statistical summaries (median, P95) - **Error categorization**: Classify failures into systematic buckets (missed tables, style loss, etc.) #### 3. Reporting & Visualization - **JSON reports**: Machine-readable output for CI/CD integration - **Markdown summaries**: Human-friendly reports with trend analysis - **Per-document breakdown**: Detailed scores for each file with failure reasons - **Comparison mode**: Side-by-side analysis of baseline vs. current runs #### 4. Validation Gates - **Hard gates**: Fail evaluation if Markdown is invalid or crashes occur - **Soft gates**: Warn if quality drops below threshold but allow pass - **Trend analysis**: Flag regressions even if absolute score passes - **Baseline anchoring**: Detect performance slowdowns vs. previous runs ### Integration Points #### With Cargo Testing ```bash # Embed metrics in standard cargo test output cargo test -p edgequake-pdf -- --nocapture # Fail CI if composite score below threshold cargo test -p edgequake-pdf --features ci-strict ``` #### With Python Evaluation Scripts ```python from pdf_validator import PDFValidator validator = PDFValidator( pdf_dir="test-data/real_dataset", gold_dir="test-data/gold", metrics=["table", "style", "robustness", "performance"] ) score = validator.evaluate() print(f"Composite Score: {score.composite}/100") ``` #### With CI/CD Pipelines ```yaml # GitHub Actions example - name: Validate PDF → Markdown run: | cargo run -p edgequake-pdf --example real_dataset_eval -- --metrics python .github/skills/pdf-markdown-validator/scripts/validate.py \ --ci-mode --fail-below 75 ``` ## Metric Definitions & Examples ### Table Accuracy Example **Input PDF:** 2×3 table with headers "Name, Age" and row "John, 25" **Gold Markdown:** ```markdown | Name | Age | | ---- | --- | | John | 25 | ``` **Generated Markdown (Perfect):** ```markdown | Name | Age | | ---- | --- | | John | 25 | ``` **Scores:** - TableDetectionF1: 1.0 (table detected correctly) - CellContentAccuracy: 1.0 (all cells match token-for-token) - TableAccuracy: **1.0** **Generated Markdown (Partial Match):** ```markdown | Name | Age | | ---- | ---- | | John | 25.0 | ``` **Scores:** - TableDetectionF1: 1.0 (table structure preserved) - CellContentAccuracy: 0.8 (minor token variations: "25" vs "25.0") - TableAccuracy: **0.9** ### Style Accuracy Example **Gold Markdown:** ```markdown # Main Heading This is **bold** and _italic_ text. ## Sub Heading More content here. ``` **Generated Markdown (Perfect):** ```markdown # Main Heading This is **bold** and _italic_ text. ## Sub Heading More content here. ``` **Scores:** - BoldF1: 1.0 - ItalicF1: 1.0 - HeadingF1: 1.0 - StyleAccuracy: **1.0** **Generated Markdown (Partial):** ```markdown # Main Heading This is bold and italic text. ## Sub Heading More content here. ``` **Scores:** - BoldF1: 0.5 (missed "bold" marker) - ItalicF1: 0.5 (missed "italic" marker) - HeadingF1: 1.0 (headings correct) - StyleAccuracy: **0.67** ### Robustness Example **Test corpus:** 30 PDFs (including 5 edge cases: corrupted, multilingual, scanned, etc.) **Results:** - Crash-free rate: 29/30 = 96.7% (1 crash on scanned PDF) - Markdown validity: 30/30 = 100% (all outputs valid) - Completeness: 28/30 = 93.3% (2 near-empty outputs from obfuscated fonts) **Robustness Score: (96.7 + 100 + 93.3) / 3 = 96.7%** ### Performance Example **Baseline (previous release):** - Median: 250ms per 1-page PDF - P95: 800ms per 1-page PDF **Current run:** - Median: 280ms per 1-page PDF - P95: 750ms per 1-page PDF **Scores:** - Median: min(1.0, 250/280) = 0.89 - P95: min(1.0, 800/750) = 1.0 - Performance: 0.5 × 0.89 + 0.5 × 1.0 = **0.95** ## Workflow: Running a Full Validation ### Step 1: Prepare Test Data ```bash # Navigate to PDF crate cd edgequake/crates/edgequake-pdf # Ensure ground-truth annotations exist # Files should be named: <pdf_name>.gold.md ls -1 test-data/real_dataset/*.gold.md ``` ### Step 2: Generate Markdown Output ```bash # Convert all PDFs to Markdown cargo run -p edgequake-pdf --example real_dataset_eval -- --write # Outputs written to: test-data/real_dataset/*.md ``` ### Step 3: Run Validation ```bash # Compute all metrics python3 ../../.github/skills/pdf-markdown-validator/scripts/validate.py \ --pdf-dir test-data/real_dataset \ --gold-dir test-data/real_dataset \ --output-report validation_report.json \ --verbose ``` ### Step 4: Analyze Results ```bash # View summary cat validation_report.json | jq '.summary' # Detailed per-document breakdown cat validation_report.json | jq '.documents | .[] | {name, scores}' # Identify failure patterns python3 ../../.github/skills/pdf-markdown-validator/scripts/analyze_failures.py \ validation_report.json --group-by failure_type ``` ### Step 5: Iterative Improvement 1. **Identify low-scoring dimensions** (table, style, etc.) 2. **Focus on highest-impact directory** (processor, renderer, layout) 3. **Implement targeted fix** with tests 4. **Re-run validation** to measure improvement 5. **Record baseline** for next comparison ## Ground-Truth Annotation Format The `.gold.md` files serve as reference implementations. Use this structure: ```markdown # Document Title (H1) ## Section Heading (H2) This paragraph contains **bold text** and _italic text_ and **_bold-italic text_**. ### Subsection (H3) #### Sub-subsection (H4)