parametric-analysis

name	parametric-analysis
description	Compute summary statistics over a text corpus — average word length, words/doc, sentences/doc, lexical diversity, readability scores, token length distributions. Use when the user wants a quantitative description of the corpus shape rather than its content.

Parametric Analysis

Descriptive stats, not semantics.

Standard metrics

• Per document: word count, sentence count, avg word length, avg sentence length, unique-token ratio (type-token ratio), reading-ease score (Flesch, Flesch-Kincaid grade).
• Corpus-wide: total docs, total tokens, vocabulary size, Zipf distribution, stopword ratio, punctuation density.
• Distributions: histograms of doc length (tokens, words, sentences), percentiles (p10/p50/p90/p99).
• Lexical diversity: MTLD or MATTR (better than raw TTR which is length-sensitive).

Procedure

1. Tokenize via spaCy (sentence + word boundaries).
2. Per-doc compute all metrics; write to a DataFrame.
3. Aggregate: corpus totals, means, medians, distributions.
4. Readability: use textstat library — Flesch, Flesch-Kincaid, Gunning Fog, SMOG, Coleman-Liau.
5. Output:
   • per-doc-metrics.csv
   • corpus-summary.json with all aggregate stats.
   • Optional histograms (matplotlib).

Integration

• Feed per-doc metrics to correlation-analysis to correlate length/readability with metadata or categories.
• Flag outliers (e.g. p99 doc length) — they often indicate junk or accidentally-concatenated docs.

No LLM. No cost. spaCy + textstat + pandas.