raw-to-knowledge-playbook

Runs the right deterministic extraction lane per office/email format and records each format's KNOWN losses UNDER A TEXT/CSV LANE in a coverage ledger so a faithful extract never masquerades as complete. Use when extracting docx/xlsx/msg/pdf/pptx sources whose richest content (formulas, charts, diagrams, attachments, images, speaker notes) does not survive a text/CSV dump.

Extracts a compressed archive (RAR/ZIP/7z) and then proves the extraction is byte-faithful by re-hashing every extracted file against the container's stored per-entry checksums — because an open-source decoder can report success while silently writing wrong bytes at the correct size (presence and size are NOT fidelity). Falls back across decoders (7z, bsdtar, official unrar) for entries one engine mis-decodes, and reports the exact set of files that no available decoder can reproduce. Use when extracting any archive before its contents enter an ingestion pipeline, especially RAR4 archives or when only partial decoders (p7zip, libarchive) are installed.

Scores committed extracts against an independent extraction engine (a different codebase than the pipeline lane) to produce measured per-lane precision/recall, with two-tier numeric scoring and a hard rule that every sub-1.0 number is attributed to a named cause. Use when a conversion lane needs correctness evidence beyond the fidelity verify loop — turning "we believe the extracts are faithful" into per-lane measured numbers.

Runs an independent adversarial verification loop on a produced change (extraction code, corpus batch, or page set) until a clean PASS — the producer never self-certifies, the reviewer is prompted to refute and to build runnable reproducers for code findings, and every fix is gated on re-running the reviewer's own reproducer. Use when a change must earn justified confidence before merge, especially pipeline-code changes where a plausible fix can be narrower than the defect class.

Independently reviews a batch of source-extract reference pages against the source binary so that no prose claims more than the committed extract supports — checking hash match, extract fidelity, and that every quoted figure traces to a committed extract with a cross-reference treated as not-a-quote and a derived number as not-quoted. Use when reviewing extracted/source-derived pages before merge, especially text-faithful pages whose prose may overreach.

Classifies XLS/XLSX workbooks before extraction and requires a traceable input data or logic contract, code artifact, and verified output artifact for a ten-file canary. Use when spreadsheet files may contain formulas, named ranges, charts, cached values, protection, or mixed data and calculation logic.

Triages a raw company archive into an ingest set by classifying on content (not extension), value-ranking candidates, deduping superseded versions, and hard EXCLUDING PII and third-party-confidential material before any extraction. Use when starting an ingestion campaign whose source is a messy company repo or shared drive full of code, build/venv, agent, and marketing noise.

Fetches the real bytes behind off-disk Git LFS pointer files via the LFS Batch API when git-lfs and sudo are unavailable, using a host token, and verifies the download because the content sha256 IS the pointer oid. Use when a source binary in a Git repo is a stored LFS pointer (a tiny version-oid-size stub) and you must read the actual file to extract from it, without installing git-lfs.

Ships an ingestion campaign as one PR per batch on STACKED branches (each off the prior tip) to avoid conflicts on shared files like an auto-generated index, and guards the merge-cascade hazard where a top stacked PR merges into an already- merged intermediate branch and strands content off main. Use when landing many sequential ingestion batches that all touch a few shared files.

Maintains an anchored-text feedback inbox for a knowledge store where every item attaches to an exact text anchor, carries an explicit resolution state, and is never silently deleted. Use when collecting, triaging, or resolving review feedback on extracted or generated pages.

Enforces the structural contract for a knowledge-store page — layered input/output split, provenance frontmatter, trust labels on every value-bearing artifact, and the public/private abstraction gate. Use when creating or reviewing any wiki/knowledge page before it is committed.

Enforces the firewall between a public knowledge store and per-client private stores by checking declared visibility, abstracting client/project identifiers by default, and running an independent publish-time grep against a maintained identifier list. Use before committing or publishing any page, and as a pre-commit/CI gate.

Runs doc-type-aware deterministic extraction on a source and makes shallow extraction visible by recording an extraction_estimate before and an extraction_yield after, flagging any source whose yield falls short. Use when ingesting a new source file (PDF/DOCX/XLSX/HTML/scanned) into the provisional store.

Vision-verifies the next batch of auto-extracted tables against rendered page images and writes closed-set verdicts with binary-faithful queue I/O, one PR per batch. Use when promoting provisional table extractions to a trusted (verified) state, correcting flagged rows, or checking no-csv figures.