Run any Skill in Manus with one click

source-extraction-coverage

Stars1

Forks2

UpdatedJune 8, 2026 at 09:01

Runs doc-type-aware deterministic extraction on a source and makes shallow extraction visible by recording an extraction_estimate before and an extraction_yield after, flagging any source whose yield falls short. Use when ingesting a new source file (PDF/DOCX/XLSX/HTML/scanned) into the provisional store.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

vamseeachanta

vamseeachanta/raw-to-knowledge-playbook

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

2 files

SKILL.md

readonly

name	source-extraction-coverage
description	Runs doc-type-aware deterministic extraction on a source and makes shallow extraction visible by recording an extraction_estimate before and an extraction_yield after, flagging any source whose yield falls short. Use when ingesting a new source file (PDF/DOCX/XLSX/HTML/scanned) into the provisional store.
license	CC-BY-4.0
compatibility	Requires a deterministic extractor per doc type (pdfplumber/Docling, python-docx, openpyxl, OCR) and a landing-page frontmatter store
metadata	{"version":"1.0","enforcement_level":"L2","status":"template","incident_refs":"coverage-2pct,A-defects,D4","params":"path:str \| type:enum(auto,pdf,docx,xlsx,html,scanned)=auto"}

source-extraction-coverage

Template skill (doc 08, doc 01). The estimate/yield pair is the whole point: it is how the 2%-coverage failure (LLM-extracted a doc and silently produced ~2% of its text) becomes a loud, machine-checkable signal instead of a silent loss.

Trigger

Preconditions

Source lives in the off-repo read-only archive; record its sha256.
Type is detected by content/structure, not filename (incident D4: names lie about content).

Steps

Estimate first. Before extracting, declare extraction_estimate in the landing-page frontmatter: roughly how much extractable content this source holds (page count, table count, "text-heavy" vs "image-only"). This is a commitment made before you see the result, so a shortfall is undeniable.
Pick the recipe by type (use ADOPT-tier permissive tools from doc 12):
- pdf (born-digital): deterministic text+geometry (pdfplumber/Docling); pdf (scanned/image-only) → route to the scanned recipe (OCR as labeled interpretation, doc 11).
- docx: structured access (python-docx) + semantic Markdown (mammoth).
- xlsx: formula-graph read (openpyxl) / fast data read (calamine) — classify by scanned structure, not sheet names (D4).
- html: typed-element extraction; strip nav/boilerplate.
- scanned: OCR with a blur gate (OpenCV Laplacian-variance → too blurry stays provisional); OCR output is a claim, not source text.
Extract deterministically. Tools, not an LLM, do the bulk copy. (LLM bulk extraction yielded ~2% coverage — never the bulk path.)
Record yield. Write extraction_yield after extraction (pages/tables actually produced, % of estimate).
Flag shortfall. If yield << estimate, mark the source extraction: shallow and queue it for a different recipe. Do not let it pass as "done."

Verification

Frontmatter has BOTH extraction_estimate (pre) and extraction_yield (post); a validator script rejects a page missing either.
Sources flagged shallow appear in the re-extraction backlog, not the trusted set.

Cleanup

Raw source never copied into any repo; only derived parts + a sources: sha256 pointer.

Incident appendix

Rule	Why
Estimate before, yield after	The 2%-coverage loss was invisible until the pair made it loud
Classify on structure not name	D4: sheet/file names misclassify content
OCR output is a claim	doc 11: scanned text is interpretation, enters the provisional gate

More from this repository

same repository

format-coverage-ledger

vamseeachanta/raw-to-knowledge-playbook

Runs the right deterministic extraction lane per office/email format and records each format's KNOWN losses UNDER A TEXT/CSV LANE in a coverage ledger so a faithful extract never masquerades as complete. Use when extracting docx/xlsx/msg/pdf/pptx sources whose richest content (formulas, charts, diagrams, attachments, images, speaker notes) does not survive a text/CSV dump.

2026-06-271

archive-extraction-integrity

vamseeachanta/raw-to-knowledge-playbook

Extracts a compressed archive (RAR/ZIP/7z) and then proves the extraction is byte-faithful by re-hashing every extracted file against the container's stored per-entry checksums — because an open-source decoder can report success while silently writing wrong bytes at the correct size (presence and size are NOT fidelity). Falls back across decoders (7z, bsdtar, official unrar) for entries one engine mis-decodes, and reports the exact set of files that no available decoder can reproduce. Use when extracting any archive before its contents enter an ingestion pipeline, especially RAR4 archives or when only partial decoders (p7zip, libarchive) are installed.

2026-06-201

independent-oracle-validation

vamseeachanta/raw-to-knowledge-playbook

Scores committed extracts against an independent extraction engine (a different codebase than the pipeline lane) to produce measured per-lane precision/recall, with two-tier numeric scoring and a hard rule that every sub-1.0 number is attributed to a named cause. Use when a conversion lane needs correctness evidence beyond the fidelity verify loop — turning "we believe the extracts are faithful" into per-lane measured numbers.

2026-06-101

adversarial-verify-loop

vamseeachanta/raw-to-knowledge-playbook

Runs an independent adversarial verification loop on a produced change (extraction code, corpus batch, or page set) until a clean PASS — the producer never self-certifies, the reviewer is prompted to refute and to build runnable reproducers for code findings, and every fix is gated on re-running the reviewer's own reproducer. Use when a change must earn justified confidence before merge, especially pipeline-code changes where a plausible fix can be narrower than the defect class.

2026-06-091

source-extract-fidelity

vamseeachanta/raw-to-knowledge-playbook

Independently reviews a batch of source-extract reference pages against the source binary so that no prose claims more than the committed extract supports — checking hash match, extract fidelity, and that every quoted figure traces to a committed extract with a cross-reference treated as not-a-quote and a derived number as not-quoted. Use when reviewing extracted/source-derived pages before merge, especially text-faithful pages whose prose may overreach.

2026-06-091

xlsx-input-code-output-canary

vamseeachanta/raw-to-knowledge-playbook

Classifies XLS/XLSX workbooks before extraction and requires a traceable input data or logic contract, code artifact, and verified output artifact for a ten-file canary. Use when spreadsheet files may contain formulas, named ranges, charts, cached values, protection, or mixed data and calculation logic.

2026-06-091

name	source-extraction-coverage
description	Runs doc-type-aware deterministic extraction on a source and makes shallow extraction visible by recording an extraction_estimate before and an extraction_yield after, flagging any source whose yield falls short. Use when ingesting a new source file (PDF/DOCX/XLSX/HTML/scanned) into the provisional store.
license	CC-BY-4.0
compatibility	Requires a deterministic extractor per doc type (pdfplumber/Docling, python-docx, openpyxl, OCR) and a landing-page frontmatter store
metadata	{"version":"1.0","enforcement_level":"L2","status":"template","incident_refs":"coverage-2pct,A-defects,D4","params":"path:str \| type:enum(auto,pdf,docx,xlsx,html,scanned)=auto"}

source-extraction-coverage

Template skill (doc 08, doc 01). The estimate/yield pair is the whole point: it is how the 2%-coverage failure (LLM-extracted a doc and silently produced ~2% of its text) becomes a loud, machine-checkable signal instead of a silent loss.

Trigger

Preconditions

Source lives in the off-repo read-only archive; record its sha256.
Type is detected by content/structure, not filename (incident D4: names lie about content).

Steps

Estimate first. Before extracting, declare extraction_estimate in the landing-page frontmatter: roughly how much extractable content this source holds (page count, table count, "text-heavy" vs "image-only"). This is a commitment made before you see the result, so a shortfall is undeniable.
Pick the recipe by type (use ADOPT-tier permissive tools from doc 12):
- pdf (born-digital): deterministic text+geometry (pdfplumber/Docling); pdf (scanned/image-only) → route to the scanned recipe (OCR as labeled interpretation, doc 11).
- docx: structured access (python-docx) + semantic Markdown (mammoth).
- xlsx: formula-graph read (openpyxl) / fast data read (calamine) — classify by scanned structure, not sheet names (D4).
- html: typed-element extraction; strip nav/boilerplate.
- scanned: OCR with a blur gate (OpenCV Laplacian-variance → too blurry stays provisional); OCR output is a claim, not source text.
Extract deterministically. Tools, not an LLM, do the bulk copy. (LLM bulk extraction yielded ~2% coverage — never the bulk path.)
Record yield. Write extraction_yield after extraction (pages/tables actually produced, % of estimate).
Flag shortfall. If yield << estimate, mark the source extraction: shallow and queue it for a different recipe. Do not let it pass as "done."

Verification

Frontmatter has BOTH extraction_estimate (pre) and extraction_yield (post); a validator script rejects a page missing either.
Sources flagged shallow appear in the re-extraction backlog, not the trusted set.

Cleanup

Raw source never copied into any repo; only derived parts + a sources: sha256 pointer.

Incident appendix

Rule	Why
Estimate before, yield after	The 2%-coverage loss was invisible until the pair made it loud
Classify on structure not name	D4: sheet/file names misclassify content
OCR output is a claim	doc 11: scanned text is interpretation, enters the provisional gate