corpus-ingest

name	corpus-ingest
description	Standardize and annotate heterogeneous writing samples (PhD theses, private notes, emails, X/Chinese posts, supervisor early-career papers) into paragraph-level analyzable units with metadata. Trigger when the user wants to ingest, preprocess, tag, or annotate corpus material for the tone-extraction workflow, or mentions corpus types A/B/C/D, paragraph segmentation, or section-function labeling.

corpus-ingest

LAYER 1 · SKILL · deterministic preprocessing. The foundation of the whole pipeline — if this is sloppy, everything downstream is wrong.

Purpose

Turn four kinds of heterogeneous writing into clean, paragraph-level units with consistent metadata so move-detector and fingerprint-miner can consume them.

Corpus types (the corpus_type tag — analysis logic differs per type)

• A — External PhD theses (implementation science / health services research / public health). Prioritize theses by students the user's supervisor has supervised. → used to extract academic tone moves.
• B — Author's pure private writing (meta-thinking notes, private reasoning; NOT reader-adjusted, NOT AI-assisted, NOT publication-polished). → purest author fingerprint / voice ground truth.
• C — Author's semi-formal & formal writing (emails to supervisor/collaborators, Master's thesis, X long-form & short-form Chinese, client advisory emails). → how the author's fingerprint deforms across registers.
• D — The supervisor's PhD-era writing (their PhD thesis, their first-author papers from candidature & just-after). Must be EARLY/CONTEMPORANEOUS work, not their current mature output. → supervisor style prototype, with characteristic flaws explicitly preserved.

Procedure

1. Read each source file. Keep types separate.
2. Segment into paragraphs.
3. Tag every paragraph with metadata:
   • corpus_type: A | B | C | D
   • register: spoken | x-chinese | email | academic
   • language
   • source: filename / citation / identifier
   • section_function: one of — topic-establishment | evidence-presentation | synthesis | scoping | transition | meta-reflection
4. section_function labeling is the priority. Tone moves attach to paragraph function; the same syntactic pattern is a different move in a scoping paragraph vs a synthesis paragraph. Spend judgment here.
5. For Chinese Type B/C material: keep the original text, AND additionally extract "semantic fingerprint features" (syntactic patterns, argument patterns — these are cross-lingually stable). Do NOT translate.

Output

A set of paragraph units, each carrying full metadata, ready for move-detector (Type A) and fingerprint-miner (Type B/C/D). Storage layout is yours to decide.

Hard constraints

• RL-3 (no averaging): preserve each paragraph as a concrete instance in its concrete context. Never collapse paragraphs into "this kind of paragraph generally."
• Do not "clean up" or correct Type B/C/D text. Errors, awkward phrasings, and idiosyncrasies are signal, not noise — especially in Type D (supervisor flaws are the carrier of resonance, see voice-synthesizer §6.3).

Self-assessment for scripting

After a representative run, judge whether any step here is mechanical and high-volume enough to deserve a deterministic Python helper (e.g. paragraph segmentation, metadata schema validation, batch file loading, syntactic-feature extraction). If so, follow the rule in SCRIPTING-POLICY.md before building it. Section-function labeling and flaw-preservation judgments must stay LLM-driven.