| name | corpus-ingest |
| description | Standardize and annotate heterogeneous writing samples (PhD theses, private notes, emails, X/Chinese posts, supervisor early-career papers) into paragraph-level analyzable units with metadata. Trigger when the user wants to ingest, preprocess, tag, or annotate corpus material for the tone-extraction workflow, or mentions corpus types A/B/C/D, paragraph segmentation, or section-function labeling. |
corpus-ingest
LAYER 1 · SKILL · deterministic preprocessing. The foundation of the whole pipeline — if this is sloppy, everything downstream is wrong.
Purpose
Turn four kinds of heterogeneous writing into clean, paragraph-level units with consistent metadata so move-detector and fingerprint-miner can consume them.
Corpus types (the corpus_type tag — analysis logic differs per type)
- A — External PhD theses (implementation science / health services research / public health). Prioritize theses by students the user's supervisor has supervised. → used to extract academic tone moves.
- B — Author's pure private writing (meta-thinking notes, private reasoning; NOT reader-adjusted, NOT AI-assisted, NOT publication-polished). → purest author fingerprint / voice ground truth.
- C — Author's semi-formal & formal writing (emails to supervisor/collaborators, Master's thesis, X long-form & short-form Chinese, client advisory emails). → how the author's fingerprint deforms across registers.
- D — The supervisor's PhD-era writing (their PhD thesis, their first-author papers from candidature & just-after). Must be EARLY/CONTEMPORANEOUS work, not their current mature output. → supervisor style prototype, with characteristic flaws explicitly preserved.
Procedure
- Read each source file. Keep types separate.
- Segment into paragraphs.
- Tag every paragraph with metadata:
corpus_type: A | B | C | Dregister: spoken | x-chinese | email | academiclanguagesource: filename / citation / identifiersection_function: one of — topic-establishment | evidence-presentation | synthesis | scoping | transition | meta-reflection
section_function labeling is the priority. Tone moves attach to paragraph function; the same syntactic pattern is a different move in a scoping paragraph vs a synthesis paragraph. Spend judgment here.- For Chinese Type B/C material: keep the original text, AND additionally extract "semantic fingerprint features" (syntactic patterns, argument patterns — these are cross-lingually stable). Do NOT translate.
Output
A set of paragraph units, each carrying full metadata, ready for move-detector (Type A) and fingerprint-miner (Type B/C/D). Storage layout is yours to decide.
Hard constraints
- RL-3 (no averaging): preserve each paragraph as a concrete instance in its concrete context. Never collapse paragraphs into "this kind of paragraph generally."
- Do not "clean up" or correct Type B/C/D text. Errors, awkward phrasings, and idiosyncrasies are signal, not noise — especially in Type D (supervisor flaws are the carrier of resonance, see voice-synthesizer §6.3).
Self-assessment for scripting
After a representative run, judge whether any step here is mechanical and high-volume enough to deserve a deterministic Python helper (e.g. paragraph segmentation, metadata schema validation, batch file loading, syntactic-feature extraction). If so, follow the rule in SCRIPTING-POLICY.md before building it. Section-function labeling and flaw-preservation judgments must stay LLM-driven.
