| name | source-intake |
| description | Ingests the operator's raw market voice (a CSV or a folder of .md/.txt files), normalizes it to the canonical input contract, indexes every source, and computes corpus sufficiency. Captures speaker_type and source_type at import so they are never guessed later. Triggers on: source intake, ingest my reviews, normalize sources, index my market voice, import my CSV of comments, build the source index, check if my corpus is thin. |
Source Intake
the first stage of a market voice run. it takes whatever raw voice the operator brings and turns it into an indexed, traceable corpus the rest of the pipeline can stand on. nothing downstream is trustworthy if intake is sloppy, because every claim eventually resolves back to a file this stage puts in raw/.
one rule above all others: capture speaker_type and source_type at import time. reconstructing who spoke later is exactly the guessing the kit exists to eliminate. if the operator's data carries those columns, keep them. if it does not, ask or default with a confidence downgrade. never infer the speaker from the text after the fact.
When to use
use when the operator has raw market voice and needs it ingested and indexed before extraction. invocable on its own (operator just wants a clean source index) or as Stage 1 of the frontrun orchestrator.
do not use to extract evidence cards or write the voice map: that is voice-extractor.
Inputs
-> a CSV following the canonical input contract, or a folder of .md/.txt files (one source per file or clearly delimited).
-> the confirmed run base path and <market-slug> from the orchestrator's Stage 0, or ask the operator if running standalone.
-> the scope facts from intake: market, business type, competitors, audience, goal.
Canonical input contract
the only hard-required column is verbatim. everything else degrades gracefully with defaults and a confidence note.
source_id (auto) · verbatim (REQUIRED) · speaker_type (rec) · source_type (rec)
· platform · source_url · date · rating · speaker_label
· is_public (public|private|anonymized) · notes
speaker_type taxonomy and source_type taxonomy live in references/source-types.md. read it before mapping columns. what makes a row a usable receipt (and when to downgrade confidence) lives in references/receipt-quality-rules.md.
Procedure
1. Confirm the run folder
confirm the base path. write nothing until it is confirmed. create:
frontrun-runs/YYYY-MM-DD-<market-slug>/
raw/
2. Normalize
run the normalizer on the operator's input:
python scripts/normalize_sources.py <input-csv-or-folder> --out frontrun-runs/YYYY-MM-DD-<market-slug>/
it produces a canonical CSV and the populated raw/ folder, and seeds 01-source-index.md. if the script is not yet present in this install, do the normalization by hand against the canonical contract: assign src_001 ascending, write one excerpt file per source into raw/ named <source>-<type>-<id>.md, and build the index manually. either way the output shape is identical.
map the operator's columns to the canonical contract. when a recommended column is missing, apply the default from source-types.md and record the downgrade in 00-intake.md.
3. Populate raw/
every source gets an unpolished excerpt file in raw/. this forces the market's actual language to stay visible and gives the traceability chain its anchor. naming: <source>-<type>-<id>.md or the imported file's own name. one source file may later yield multiple evidence cards, all pointing back to its src_id.
4. Write 00-intake.md
scope: market/category, business type, sources in and out of scope, competitors, audience, confidence limits, and what this run should produce. if a public hard number is being claimed (e.g. total conversation count), the exact traceable figure lives here, nowhere more public.
5. Write 01-source-index.md
one row per source: src_id, source_type, platform, url/path, date, reliability (from the receipt-quality rules), and is_public as public|private|anonymized. this index is the middle link of the traceability chain: evidence_id -> card -> source_id -> this row -> file in raw/. a row with no raw/ file is a broken chain and must not exist.
6. Compute corpus sufficiency and write 00b-scope-contract.md
the corpus-sufficiency gate. record:
-> source_count (total indexed sources)
-> distinct_source_types (how many of the taxonomy are represented)
-> recency window (oldest to newest date)
-> thin_pool (boolean)
-> corpus_limitation_note (required whenever thin_pool == true, plain language)
thin_pool is true when source_count is below threshold OR distinct_source_types is below threshold. canonical thresholds (the same numbers validate_run.py enforces): fewer than 12 sources, or fewer than 3 distinct source types. when thin, say so plainly in the note. the orchestrator caps the angle slate at 3 and the validator enforces it. never launder thin comment noise into survey math (no "73% of customers" from 40 comments).
Outputs
| File | What it carries |
|---|
00-intake.md | scope, business type, sources in/out, competitors, audience, confidence limits, goal |
00b-scope-contract.md | source_count, distinct_source_types, recency window, thin_pool, corpus_limitation_note |
01-source-index.md | every source: src_id, source_type, platform, url/path, date, reliability, is_public |
raw/ | one unpolished excerpt file per source. the anchor of every traceability chain. |
References
references/source-types.md: the source_type and speaker_type taxonomies and how each maps to reliability.references/receipt-quality-rules.md: what makes a usable receipt, when to downgrade confidence, and is_public handling.
