extract-tables

> STOP. READ THIS ENTIRE SKILL.MD BEFORE CALLING ANY ENDPOINT. # /extract-tables Composable, self-correcting PDF table extraction. Replaces Camelot with a hybrid Rust + compiled-Python architecture for 5-15x speed and 5-8x memory improvement. ## Quick Start ```bash # Extract tables from a PDF ./run.sh extract document.pdf --output tables.json # Extract with specific strategy ./run.sh extract document.pdf --strategy lattice --line-scale 15 # Extract with auto-strategy (Shadow-LEGO routing) ./run.sh extract document.pdf --auto # Batch extraction ./run.sh batch /path/to/pdfs/ --output-dir /path/to/results/ # Python API (drop-in replacement for camelot.io.read_pdf) python -c "from extract_tables import read_pdf; tables = read_pdf('doc.pdf')" # Health check ./sanity.sh ``` ## Architecture ``` PDF input | v [pdf_oxide] ---- Rust: text extraction + layout analysis (replaces pdfminer) | v [Strategy Router] -- Shadow-LEGO 3-tier cascade: | T0: Heuristic (line density, text gaps) | T0.5: /assistant classify(task="table-strategy") | T2: /scillm teacher (novel layouts) v [Parser] ----------- Compiled Python (mypyc): | - Lattice: calls Rust image_proc for OpenCV ops | - Stream: pure algorithm (text positioning) | - Network: alignment-based detection | - Hybrid: lattice + stream combination v [Rust image_proc] -- PyO3 module (replaces OpenCV Python): | - adaptive_threshold | - morphological ops (erode, dilate) | - find_contours | - find_lines (Hough transform) v [Table Construction] Compiled Python: cell assignment, spanning cells | v [polars DataFrame] - Output (replaces pandas) | v [Shadow Logger] ---- Self-correction: log quality, compare strategies ``` ## Shadow-LEGO Strategy Routing The strategy router follows the 3-tier cascade pattern: | Tier | Component | Cost | Latency | |------|-----------|------|---------| | T0 | Heuristic: line density > 0.3 → lattice, else stream | Free | <1ms | | T0.5 | `/assistant classify(task="table-strategy-router")` | Free | ~10ms | | T1.5 | `/create-table-classifier` MobileNetV2 prediction | Free | ~50ms | | T2 | `/scillm` teacher validation (novel layouts only) | Paid | ~2s | Escalation stops at first confident tier (confidence >= 0.7). ## Self-Correction Loop ``` 1. Extract tables with predicted strategy 2. Compute quality metrics (accuracy, whitespace, fragmentation) 3. If quality < threshold: a. Log disagreement to shadow.jsonl b. Try alternative strategy c. Compare results d. Learn: update /memory with winning parameters 4. Nightly: /table-lab tunes parameters from shadow logs ``` ## Rust Crate (PyO3) ``` src/rust/ ├── Cargo.toml ├── src/ │ ├── lib.rs # PyO3 module entry │ ├── image_proc.rs # Threshold, morphology, contours │ ├── geometry.rs # Line merging, bbox math, scaling │ └── pdf_bridge.rs # pdf_oxide integration ``` Built with maturin. Exposes: - `extract_tables.rust.adaptive_threshold(image_path) -> (img, threshold)` - `extract_tables.rust.find_lines(threshold, direction, line_scale) -> lines` - `extract_tables.rust.find_contours(vertical, horizontal) -> contours` - `extract_tables.rust.find_joints(contours, vertical, horizontal) -> joints` - `extract_tables.rust.merge_close_lines(lines, tolerance) -> merged` - `extract_tables.rust.scale_coordinates(coords, scalers) -> scaled` ## Python API (drop-in Camelot replacement) ```python from extract_tables import read_pdf # Same interface as camelot.io.read_pdf tables = read_pdf( "document.pdf", pages="1", flavor="lattice", # or "stream", "network", "hybrid", "auto" line_scale=15, process_background=False, ) # Each table has: table = tables[0] table.df # polars DataFrame table.accuracy # float table.whitespace # float table.page # int table.order # int table.bbox # (x1, y1, x2, y2) # Export table.to_csv("output.csv") table.to_json("output.json") table.to_excel("output.xlsx") tables.export("output.csv", f="csv", compress=True) ``` ## Extractor Integration The `/extractor` pipeline (s05_table_extractor.py) calls: ```python # Before (Camelot): from camelot import io as camelot_io tables = camelot_io.read_pdf(str(pdf_path), pages=page_str, flavor=flavor, **params) # After (extract-tables): from extract_tables import read_pdf tables = read_pdf(str(pdf_path), pages=page_str, flavor=flavor, **params) ``` ## Dependencies ### Rust (compiled into PyO3 wheel) - pdf_oxide — PDF text extraction + layout - opencv-rust or imageproc — Image processing - ndarray — Array operations - pyo3 + maturin — Python bindings ### Python (compiled with mypyc) - polars — DataFrame output - click — CLI ### Docker (optional, for ghostscript/poppler backends) - ghostscript — Alternative PDF rendering - poppler-utils — pdftoppm rendering ## Metrics | Metric | Description | |--------|-------------| | accuracy | % of text correctly placed in cells | | whitespace | % of empty space in table | | fragmentation | Score indicating over-split cells | | strategy_confidence | Shadow-LEGO routing confidence | ## Files ``` ~/.claude/skills/extract-tables/ ├── SKILL.md # This file ├── run.sh # Entry point ├── sanity.sh # Health check ├── src/ │ ├── rust/ # Rust crate (PyO3) │ │ ├── Cargo.toml │ │ └── src/ │ │ ├── lib.rs │ │ ├── image_proc.rs │ │ ├── geometry.rs │ │ └── pdf_bridge.rs │ └── python/ │ ├── __init__.py │ ├── extract_tables.py # Main API: read_pdf() │ ├── strategy_router.py # Shadow-LEGO strategy selection │ ├── table.py # Table/Cell data structures │ ├── metrics.py # Accuracy, whitespace, fragmentation │ ├── export.py # CSV, JSON, Excel, HTML, Markdown │ └── parsers/ │ ├── __init__.py │ ├── base.py │ ├── lattice.py │ ├── stream.py │ ├── network.py │ └── hybrid.py ├── docker/ │ ├── Dockerfile.ghostscript │ └── Dockerfile.poppler ├── shadow.jsonl # Self-correction logs ├── tests/ │ ├── fixtures/ # PDF test files │ └── test_extract.py └── sanity/ └── check_deps.sh ```