readability-lexical-diversity

name	readability-lexical-diversity
description	Use when computing quantitative text complexity metrics from a corpus, measuring readability scores (Flesch-Kincaid, Coleman-Liau, Gunning Fog, SMOG, ARI), lexical diversity (TTR, MTLD, MATTR, hapax legomena), or translating complexity measurements into LLM prompt constraints such as target grade levels and vocabulary diversity ranges

Readability and Lexical Diversity Metrics

Overview

Compute quantitative complexity targets from a text corpus by measuring readability (grade-level estimates from multiple formulas) and lexical diversity (vocabulary richness corrected for text length). The core principle: no single readability formula or diversity index is sufficient -- use multiple measures, report the range, and correct for text length before comparing. Readability formulas estimate the US grade level required to comprehend text; lexical diversity indices estimate how varied the vocabulary is. Both translate directly into measurable LLM prompt constraints (e.g., "target FK grade 12-14, MTLD above 80, average sentence length 18-22 words").

Why multiple formulas: Flesch-Kincaid and Coleman-Liau can diverge by 2-4 grade levels on the same text because they weight different proxies for complexity (syllables vs. characters, sentence length vs. word length). Reporting only one formula hides this disagreement. The consensus range across formulas is the reliable signal.

When to Use

• Measuring the complexity profile of a text corpus before voice replication
• Establishing quantitative readability targets for LLM prompt constraints
• Comparing complexity across documents, authors, or time periods within a corpus
• Computing lexical diversity with length-corrected measures (MTLD, MATTR)
• Setting a complexity ceiling and floor for generated text
• Feeding readability/diversity metrics into an archetype or persona construction pipeline

When NOT to use:

• Texts shorter than 100 words (readability formulas are unreliable; see Insufficient Data Handling)
• Comparing readability scores across languages without recalibration (formulas are English-calibrated)
• Equating readability with writing quality (low grade level is not "bad"; high grade level is not "good")
• Claiming readability scores measure actual reader comprehension (they estimate surface complexity)
• Using raw TTR on variable-length texts without correction (TTR decreases mechanically as text length increases)

digraph readability_decision {
    rankdir=TB;
    "Texts 100+ words?" [shape=diamond];
    "English text?" [shape=diamond];
    "Variable text lengths\nin corpus?" [shape=diamond];
    "Full analysis:\nmultiple formulas +\nlength-corrected LD" [shape=box];
    "Report insufficient\ndata per formula" [shape=box];
    "Flag: formulas not\ncalibrated for this language" [shape=box];
    "Use MTLD/MATTR\nnot raw TTR" [shape=box];

    "Texts 100+ words?" -> "English text?" [label="yes"];
    "Texts 100+ words?" -> "Report insufficient\ndata per formula" [label="no"];
    "English text?" -> "Variable text lengths\nin corpus?" [label="yes"];
    "English text?" -> "Flag: formulas not\ncalibrated for this language" [label="no"];
    "Variable text lengths\nin corpus?" -> "Use MTLD/MATTR\nnot raw TTR" [label="yes, variable"];
    "Variable text lengths\nin corpus?" -> "Full analysis:\nmultiple formulas +\nlength-corrected LD" [label="no, uniform"];
}

Quick Reference

Readability Formulas

Formula	What It Measures	Minimum Text	Key Sensitivity	Python (textstat)
Flesch-Kincaid Grade Level	Syllables per word + words per sentence	100 words	Sentence length, polysyllabic words	textstat.flesch_kincaid_grade(text)
Coleman-Liau Index	Characters per word + sentences per 100 words	100 words	Word length in characters (not syllables)	textstat.coleman_liau_index(text)
Gunning Fog Index	Complex words (3+ syllables) + sentence length	100 words	Polysyllabic word frequency	textstat.gunning_fog(text)
SMOG Index	Polysyllabic words (3+ syllables)	30 sentences (~600 words)	Polysyllabic density; needs 30-sentence sample	textstat.smog_index(text)
Automated Readability Index (ARI)	Characters per word + words per sentence	100 words	Character counts (fast, no syllable parsing)	textstat.automated_readability_index(text)

Lexical Diversity Indices

Index	What It Measures	Length-Sensitive?	Correction Method	Python Library
TTR (Type-Token Ratio)	Unique words / total words	YES -- decreases with length	Do not compare across different-length texts	len(set(tokens)) / len(tokens)
MTLD	Average consecutive words maintaining TTR >= 0.72	NO (r = -0.02 with length)	Built-in; robust to length	lexical_diversity.mtld(tokens)
MATTR	Moving-average TTR across fixed windows (default 50 tokens)	Minimal (window-based)	Fixed window normalizes length	lexical_diversity.mattr(tokens, window=50)
HD-D	Hypergeometric distribution of word sampling	Minimal	Probability-based; sample-size aware	lexical_diversity.hdd(tokens)
Hapax Legomena Ratio	Words appearing exactly once / total words	YES -- decreases with length	Report alongside MTLD/MATTR, not alone	sum(1 for w,c in counts.items() if c==1) / len(tokens)

Grade Level Interpretation

Grade Level	Audience	Example Register
5-6	General public, plain language	Newspaper articles, patient information
7-8	Educated general audience	Popular science, quality journalism
9-10	High school level	Magazine features, opinion columns
11-12	College-ready	Specialized journalism, policy briefs
13-14	Undergraduate level	Academic introductions, technical blogs
15-16	Graduate/professional	Academic papers, legal documents
17+	Specialist	Dense academic prose, statutory language

Minimum Text Length Per Formula

Formula	Hard Minimum	Recommended Minimum	Below Minimum Action
Flesch-Kincaid	100 words	200+ words	Exclude from per-document scoring; include only in corpus aggregate
Coleman-Liau	100 words	200+ words	Same
Gunning Fog	100 words	200+ words	Same
SMOG	30 sentences	600+ words	Skip SMOG for this text; use other formulas
ARI	100 words	200+ words	Same
TTR	50 tokens	Equal-length samples only	Use MTLD or MATTR instead
MTLD	100 tokens	200+ tokens	Flag as low-confidence
MATTR	50 tokens (= window size)	200+ tokens	Flag as low-confidence

Workflow

Copy this checklist and track progress:

Readability & Lexical Diversity Analysis Progress:
- [ ] Step 1: Validate corpus suitability (text lengths, language, count)
- [ ] Step 2: Compute per-document readability scores (all five formulas)
- [ ] Step 3: Compute per-document lexical diversity (MTLD, MATTR, hapax)
- [ ] Step 4: Compute corpus-level aggregate statistics
- [ ] Step 5: Analyze formula agreement and divergence
- [ ] Step 6: Translate metrics into LLM prompt constraints
- [ ] Step 7: Write findings to docs/analysis/19-readability-lexical-diversity.md

Step 1: Validate Corpus Suitability

Before computing metrics, assess every document against minimum thresholds.

Suitability checks:

Check	Pass Condition	Fail Action
Language	Predominantly English	Non-English text invalidates all formula calibrations. Exclude or flag.
Corpus size	30+ documents	Below 30, corpus-level aggregates are unreliable. Report per-document only.
Word count per document	100+ words	Below 100: exclude from per-document scoring, include only in corpus aggregate via concatenation.
Sentence count per document	3+ sentences	Single-sentence texts produce meaningless readability scores. Exclude or concatenate.
Text length variance	Track coefficient of variation	If CV > 1.0 (highly variable lengths), raw TTR comparisons are invalid. Use MTLD/MATTR exclusively.

import textstat
import re
from collections import Counter

def assess_document_suitability(text):
    """Assess whether a document meets minimum thresholds for each formula."""
    words = text.split()
    sentences = [s.strip() for s in re.split(r'[.!?]+', text) if s.strip()]
    word_count = len(words)
    sentence_count = len(sentences)

    return {
        'word_count': word_count,
        'sentence_count': sentence_count,
        'fk_eligible': word_count >= 100,
        'cli_eligible': word_count >= 100,
        'fog_eligible': word_count >= 100,
        'smog_eligible': sentence_count >= 30,
        'ari_eligible': word_count >= 100,
        'mtld_eligible': word_count >= 100,
        'mattr_eligible': word_count >= 50,
    }

If most documents fail suitability: Consider concatenating short documents into larger units (by author, by thread, by time window) before scoring. Document the concatenation strategy and note that scores reflect the aggregate unit, not individual documents.

Step 2: Compute Per-Document Readability Scores

Score every eligible document with all five formulas. Report all five -- never choose one formula and discard the rest.

import textstat
import numpy as np

def compute_readability(text, suitability):
    """Compute all readability scores for an eligible document.
    Returns None for formulas where the document is below minimum."""
    scores = {}

    if suitability['fk_eligible']:
        scores['flesch_kincaid'] = textstat.flesch_kincaid_grade(text)
    if suitability['cli_eligible']:
        scores['coleman_liau'] = textstat.coleman_liau_index(text)
    if suitability['fog_eligible']:
        scores['gunning_fog'] = textstat.gunning_fog(text)
    if suitability['smog_eligible']:
        scores['smog'] = textstat.smog_index(text)
    if suitability['ari_eligible']:
        scores['ari'] = textstat.automated_readability_index(text)

    # Structural features (always computable if 1+ sentence)
    words = text.split()
    sentences = [s.strip() for s in re.split(r'[.!?]+', text) if s.strip()]
    scores['avg_sentence_length'] = len(words) / max(len(sentences), 1)
    scores['avg_word_length_chars'] = np.mean([len(w) for w in words])
    scores['syllables_per_word'] = textstat.avg_syllables_per_word(text)

    # Consensus grade level (median of available formula scores)
    grade_scores = [v for k, v in scores.items()
                    if k in ('flesch_kincaid', 'coleman_liau', 'gunning_fog',
                             'smog', 'ari') and v is not None]
    if grade_scores:
        scores['consensus_grade'] = np.median(grade_scores)
        scores['grade_range'] = (min(grade_scores), max(grade_scores))
        scores['grade_spread'] = max(grade_scores) - min(grade_scores)

    return scores

Critical: The consensus_grade (median across formulas) and grade_range (min-max spread) are the primary outputs. Individual formula scores are evidence; the range is the finding.

Step 3: Compute Per-Document Lexical Diversity

Use length-corrected measures (MTLD, MATTR) as primary indices. Report raw TTR only as a supplementary reference, never as a standalone comparison across different-length texts.

from collections import Counter

def compute_lexical_diversity(text, suitability):
    """Compute lexical diversity with length correction.
    MTLD and MATTR are primary; TTR and hapax are supplementary."""
    words = text.lower().split()
    word_count = len(words)
    scores = {'token_count': word_count}

    if word_count == 0:
        return scores

    # Raw TTR -- supplementary only, do NOT compare across different lengths
    unique_words = set(words)
    scores['ttr_raw'] = len(unique_words) / word_count
    scores['type_count'] = len(unique_words)

    # Hapax legomena (words appearing exactly once)
    freq = Counter(words)
    hapax_count = sum(1 for w, c in freq.items() if c == 1)
    scores['hapax_count'] = hapax_count
    scores['hapax_ratio'] = hapax_count / word_count

    # MTLD -- primary length-corrected measure
    if suitability['mtld_eligible']:
        try:
            from lexical_diversity import lex_div as ld
            scores['mtld'] = ld.mtld(words)
        except ImportError:
            scores['mtld'] = _compute_mtld_fallback(words)

    # MATTR -- moving-average TTR (window=50 tokens)
    if suitability['mattr_eligible']:
        scores['mattr_50'] = _compute_mattr(words, window=50)

    return scores

def _compute_mattr(tokens, window=50):
    """Moving-Average Type-Token Ratio.
    Calculates TTR for each sliding window and averages."""
    if len(tokens) < window:
        return len(set(tokens)) / len(tokens)  # fall back to TTR
    ttrs = []
    for i in range(len(tokens) - window + 1):
        window_tokens = tokens[i:i + window]
        ttrs.append(len(set(window_tokens)) / window)
    return np.mean(ttrs)

def _compute_mtld_fallback(tokens, threshold=0.72):
    """MTLD: average segment length before TTR drops below threshold.
    Runs forward and backward, averages both passes."""
    def _one_pass(tokens):
        factor_count = 0
        factor_length = 0
        current_start = 0
        for i in range(1, len(tokens) + 1):
            segment = tokens[current_start:i]
            ttr = len(set(segment)) / len(segment)
            if ttr <= threshold:
                factor_count += 1
                factor_length += len(segment)
                current_start = i
        # Handle partial factor at end
        if current_start < len(tokens):
            remaining = tokens[current_start:]
            remaining_ttr = len(set(remaining)) / len(remaining)
            if remaining_ttr < 1.0:
                partial = (1.0 - remaining_ttr) / (1.0 - threshold)
                factor_count += partial
                factor_length += len(remaining)
        if factor_count == 0:
            return len(tokens)  # TTR never dropped below threshold
        return factor_length / factor_count

    forward = _one_pass(tokens)
    backward = _one_pass(list(reversed(tokens)))
    return (forward + backward) / 2

Why MTLD is primary: McCarthy and Jarvis (2010) found MTLD has negligible correlation with text length (r = -0.02), while TTR correlates strongly (r = -0.70 to -0.90). MTLD reflects how many consecutive words a writer sustains before repeating vocabulary. Higher MTLD = more diverse vocabulary. Typical ranges: 50-80 (moderate), 80-120 (high), 120+ (very high).

Step 4: Compute Corpus-Level Aggregate Statistics

Aggregate per-document scores into corpus-level distributions.

import pandas as pd

def compute_corpus_aggregates(doc_scores_df):
    """Compute corpus-level statistics from per-document scores."""
    metrics = ['flesch_kincaid', 'coleman_liau', 'gunning_fog', 'smog',
               'ari', 'consensus_grade', 'avg_sentence_length',
               'mtld', 'mattr_50', 'ttr_raw', 'hapax_ratio']

    aggregates = {}
    for metric in metrics:
        if metric not in doc_scores_df.columns:
            continue
        col = doc_scores_df[metric].dropna()
        if len(col) < 2:
            continue
        aggregates[metric] = {
            'mean': col.mean(),
            'median': col.median(),
            'std': col.std(),
            'min': col.min(),
            'max': col.max(),
            'p25': col.quantile(0.25),
            'p75': col.quantile(0.75),
            'n_scored': len(col),
        }

    return aggregates

Step 5: Analyze Formula Agreement and Divergence

This step answers the critical question: do the formulas agree, and where do they disagree?

Agreement analysis:

1. For each document, compute the spread (max grade - min grade across formulas)
2. Compute the corpus-level median spread
3. Flag documents where spread exceeds 4 grade levels -- these often contain unusual structures (very long sentences with simple words, or short sentences with technical jargon)

Interpreting divergence:

Divergence Pattern	Cause	Implication
FK high, CLI low	Long sentences with short words (syllable-heavy but character-light)	Report the range, not one formula
CLI high, FK low	Short sentences with long technical terms (character-heavy but sentence structure is simple)	Coleman-Liau overweights jargon
Fog high, others low	Many polysyllabic words that are common (e.g., "important", "understanding")	Fog penalizes syllable count regardless of word familiarity
SMOG higher than FK	Dense polysyllabic vocabulary throughout	SMOG was designed for health materials; it reads polysyllabic density more strictly
All formulas agree within 2 grades	Text has consistent complexity across all proxies	High-confidence consensus

Step 6: Translate Metrics Into LLM Prompt Constraints

This is the critical output step: converting measured complexity into actionable constraints for text generation prompts.

Constraint mapping table:

Metric	Measured Value (example)	LLM Prompt Constraint
Consensus grade level	12.5 (range: 11-14)	"Write at a late high school to early college reading level (FK grade 11-14)"
Average sentence length	19.3 words	"Target 17-22 words per sentence on average; vary between 8 and 35"
Average word length	5.1 characters	"Prefer words of 4-6 characters; use longer technical terms sparingly"
MTLD	94.2	"Maintain high vocabulary diversity; avoid repeating the same word within 5 sentences unless it is a key term"
MATTR (window=50)	0.78	"Within any 50-word span, use at least 35-40 unique words"
Hapax ratio	0.52	"Approximately half of all words used should appear only once in the text"
Syllables per word	1.62	"Mix monosyllabic and polysyllabic words; average around 1.5-1.7 syllables per word"

Constraint output format (embed in prompts):

## Complexity Constraints
- **Grade level:** Target FK grade [X-Y] (consensus range from corpus analysis)
- **Sentence length:** Average [N-M] words per sentence; vary between [min] and [max]
- **Vocabulary diversity:** MTLD target [N]+; avoid repeating non-function words within [N] sentences
- **Word complexity:** Average [N] syllables per word; ~[N]% of words should be 3+ syllables
- **Lexical range:** Approximately [N]% of words should be unique (hapax target)

Translating ranges, not points: Always convert measured metrics into target ranges, not exact values. If measured FK = 12.3, the constraint should be "FK grade 11-14" (not "FK grade 12.3"). Readability scores have an inherent precision of approximately +/- 1.5 grade levels on texts of 200+ words.

Step 7: Write the Report

Write all findings to docs/analysis/19-readability-lexical-diversity.md.

Report Output Template

The final report MUST be written to docs/analysis/19-readability-lexical-diversity.md with this structure:

# Readability and Lexical Diversity Analysis

## Methodology
- **Corpus:** [N documents scored, N excluded for insufficient length, total word count]
- **Readability formulas:** Flesch-Kincaid, Coleman-Liau, Gunning Fog, SMOG, ARI (via textstat)
- **Lexical diversity:** MTLD, MATTR (window=50), raw TTR (supplementary), hapax legomena ratio
- **Minimum thresholds:** 100 words per document for readability, 30 sentences for SMOG, 100 tokens for MTLD
- **Aggregation:** Per-document scoring with corpus-level median, IQR, and range

## Corpus Suitability Assessment
- **Total documents:** [N]
- **Documents meeting 100-word minimum:** [N] ([%])
- **Documents meeting 30-sentence minimum (SMOG):** [N] ([%])
- **Language:** [English / mixed]
- **Text length distribution:** mean [N] words, median [N], std [N], range [min-max]
- **Text length CV:** [X.XX] [uniform / moderate / highly variable]
- **Overall suitability:** [suitable / suitable with caveats / unsuitable]

## Readability Scores

### Per-Formula Corpus Statistics
| Formula | Mean | Median | Std | P25 | P75 | Min | Max | N Scored |
|---------|------|--------|-----|-----|-----|-----|-----|----------|
| Flesch-Kincaid | | | | | | | | |
| Coleman-Liau | | | | | | | | |
| Gunning Fog | | | | | | | | |
| SMOG | | | | | | | | |
| ARI | | | | | | | | |

### Consensus Grade Level
- **Median consensus grade:** [X.X] (IQR: [X.X - X.X])
- **Corpus grade range:** [X - X] (min to max of per-document consensus)
- **Interpretation:** [audience level description]

### Formula Agreement Analysis
- **Median per-document spread:** [X.X] grade levels
- **Documents with spread > 4 grades:** [N] ([%])
- **Most divergent formula pair:** [Formula A] vs [Formula B] (mean difference: [X.X])
- **Interpretation:** [why the formulas disagree for this corpus]

### Structural Metrics
| Metric | Mean | Median | Std | P25 | P75 |
|--------|------|--------|-----|-----|-----|
| Avg sentence length (words) | | | | | |
| Avg word length (characters) | | | | | |
| Syllables per word | | | | | |

## Lexical Diversity

### Corpus Statistics
| Index | Mean | Median | Std | P25 | P75 | N Scored |
|-------|------|--------|-----|-----|-----|----------|
| MTLD | | | | | | |
| MATTR (w=50) | | | | | | |
| TTR (raw, supplementary) | | | | | | |
| Hapax ratio | | | | | | |

### Length Correction Verification
- **TTR-length correlation:** r = [X.XX] (should be strongly negative)
- **MTLD-length correlation:** r = [X.XX] (should be near zero)
- **MATTR-length correlation:** r = [X.XX] (should be near zero)
- **Conclusion:** [MTLD/MATTR confirmed length-independent / TTR confirmed length-dependent]

## LLM Prompt Constraints (Derived)

### Complexity Ceiling and Floor
| Metric | Floor (P25) | Target (Median) | Ceiling (P75) | Constraint |
|--------|-------------|-----------------|---------------|------------|
| Grade level | | | | "Write at FK grade [X-Y]" |
| Sentence length | | | | "Average [X-Y] words per sentence" |
| Vocabulary diversity (MTLD) | | | | "Maintain MTLD of [X]+" |
| MATTR | | | | "In any 50-word span, [X-Y]% unique words" |
| Word complexity | | | | "Average [X-Y] syllables per word" |

### Full Constraint Block (Copy-Paste Ready)
[Pre-formatted constraint block for direct embedding in LLM prompts]

## Limitations and Caveats
- Readability formulas estimate surface complexity, not comprehension difficulty
- Grade levels are approximate (+/- 1.5 grades on 200+ word texts)
- Different formulas weight different proxies; the consensus range is the reliable signal
- Lexical diversity scores reflect vocabulary range, not vocabulary quality or appropriateness
- SMOG requires 30+ sentences; documents below this threshold were excluded from SMOG scoring
- [Corpus-specific limitations from Step 1]
- Short texts (< 200 words) produce less stable scores; [N] documents were in this range
- These metrics do NOT measure: clarity, coherence, accuracy, persuasiveness, or engagement

## References
- Flesch, R. (1948). A new readability yardstick. *Journal of Applied Psychology*, 32(3), 221-233.
- Kincaid, J.P. et al. (1975). Derivation of new readability formulas for Navy enlisted personnel. (Research Branch Report 8-75). Naval Technical Training Command.
- Coleman, M. & Liau, T.L. (1975). A computer readability formula designed for machine scoring. *Journal of Applied Psychology*, 60(2), 283-284.
- Gunning, R. (1952). *The Technique of Clear Writing*. McGraw-Hill.
- McLaughlin, G.H. (1969). SMOG grading: A new readability formula. *Journal of Reading*, 12(8), 639-646.
- McCarthy, P.M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. *Behavior Research Methods*, 42(2), 381-392.
- Covington, M.A. & McFall, J.D. (2010). Cutting the Gordian knot: The moving-average type-token ratio (MATTR). *Journal of Quantitative Linguistics*, 17(2), 94-100.
- textstat Python library: https://github.com/textstat/textstat
- lexical_diversity Python library: https://github.com/kristopherkyle/lexical_diversity
- TAALED: https://github.com/LCR-ADS-Lab/TAALED

Good Patterns

• Use multiple readability formulas and report the range -- no single formula captures complexity; the consensus (median) and spread across FK, CLI, Fog, SMOG, and ARI is the reliable signal
• Correct TTR for text length using MTLD or MATTR -- raw TTR is mechanically coupled to text length and cannot be compared across documents of different sizes
• Compute per-document AND corpus-level metrics -- per-document scores show variance; corpus-level aggregates set the constraint targets
• Report grade levels as ranges, not points -- "FK 11-14" is honest; "FK 12.3" implies false precision
• Translate every metric into a concrete LLM prompt constraint -- the analysis is only useful if it produces a copy-paste-ready constraint block
• Verify length correction by computing TTR-length and MTLD-length correlations -- confirm that MTLD/MATTR are indeed length-independent for your corpus
• Establish both ceiling and floor -- complexity constraints need an upper and lower bound, not just a target
• Report the number of documents excluded and why -- transparency about data loss from minimum thresholds

Anti-Patterns

Anti-Pattern	Why It Fails	Instead
Using a single readability formula	Formulas diverge by 2-4 grades on the same text; one formula hides disagreement	Run all five; report consensus median and range
Computing raw TTR on variable-length texts	TTR decreases mechanically with text length (r = -0.70 to -0.90); longer texts always appear "less diverse"	Use MTLD (length-independent) or MATTR (window-normalized)
Treating grade levels as precise measurements	FK 12.3 vs 12.5 is noise, not signal; formulas have ~1.5 grade precision	Report as ranges (e.g., "grade 11-14")
Ignoring that short texts produce unreliable scores	Below 100 words, sentence and word sampling is too sparse; one long sentence skews the entire score	Enforce minimum thresholds per formula; exclude or concatenate short texts
Conflating readability with quality	Low grade level serves different audiences; high grade level may indicate jargon, not sophistication	Frame as "complexity targeting" not "quality assessment"
Reporting only corpus averages	Averages hide bimodal distributions and outliers	Report median, IQR, min, max, and per-document distributions
Comparing TTR across authors with different text volumes	Author with 500 words will have higher TTR than author with 5,000 words regardless of actual diversity	Use MTLD for cross-author comparison
Skipping SMOG when texts have < 30 sentences	SMOG's formula requires 30 sentences; below this, the polysyllabic sampling is unreliable	Report SMOG as N/A for short texts; use the other four formulas
Using readability as sole voice fingerprint	Readability measures surface structure only; two very different writers can have the same FK grade	Combine with sentiment, personality, and stylistic metrics

Boundaries

This skill SHOULD produce:

• Per-document readability scores from all five formulas with eligibility tracking
• Per-document lexical diversity scores (MTLD, MATTR, hapax ratio) with length correction
• Corpus-level aggregate statistics (median, IQR, range) for all metrics
• Formula agreement analysis (median spread, divergent pairs)
• Complexity ceiling and floor (P25 and P75 of consensus grade)
• A copy-paste-ready LLM prompt constraint block derived from the metrics
• Length-correction verification (TTR-length vs MTLD-length correlations)
• Written report at docs/analysis/19-readability-lexical-diversity.md

This skill should NOT:

• Treat grade levels as exact (always report ranges with +/- 1.5 grade precision)
• Compare readability across languages without noting that formulas are English-calibrated
• Claim readability scores measure actual reader comprehension (they estimate surface complexity)
• Use metrics on texts shorter than 100 words without flagging unreliability
• Present raw TTR as a valid cross-document comparison when text lengths differ
• Equate high readability scores with good writing or low scores with bad writing
• Use a single formula and discard the others
• Skip the constraint translation step (metrics without actionable constraints are incomplete)
• Produce a report without a corpus suitability assessment

Insufficient Data Handling

digraph insufficient_data {
    rankdir=TB;
    node [shape=diamond, style=filled, fillcolor="#f0f0f0"];

    start [label="Per-document\nword count?", shape=diamond];
    tiny [label="< 100 words", shape=box, fillcolor="#ffcccc"];
    short [label="100-199 words", shape=box, fillcolor="#fff3cc"];
    adequate [label="200-599 words", shape=box, fillcolor="#ffffcc"];
    good [label="600+ words", shape=box, fillcolor="#ccffcc"];

    start -> tiny [label="< 100"];
    start -> short [label="100-199"];
    start -> adequate [label="200-599"];
    start -> good [label="600+"];

    tiny_action [label="EXCLUDE from per-document scoring.\nContribute to corpus aggregate\nonly via concatenation.", shape=box, fillcolor="#ffcccc"];
    short_action [label="Score FK, CLI, Fog, ARI.\nSkip SMOG.\nFlag MTLD as low-confidence.\nWiden grade precision to +/- 2.5.", shape=box, fillcolor="#fff3cc"];
    adequate_action [label="Score FK, CLI, Fog, ARI.\nSMOG only if 30+ sentences.\nFull lexical diversity.\nStandard precision +/- 1.5.", shape=box, fillcolor="#ffffcc"];
    good_action [label="All formulas eligible\nincluding SMOG.\nFull lexical diversity.\nHighest confidence.", shape=box, fillcolor="#ccffcc"];

    tiny -> tiny_action;
    short -> short_action;
    adequate -> adequate_action;
    good -> good_action;
}

Condition	Action
Document < 100 words	Exclude from per-document readability scoring. These texts contribute to corpus aggregates only if concatenated into larger units (by author, thread, or time window). Report the number excluded.
Document 100-199 words	Score with FK, CLI, Fog, ARI. Skip SMOG (needs 30 sentences). Flag MTLD as "low confidence." Widen grade-level precision to +/- 2.5 grades.
Document 200-599 words	Full scoring except SMOG (only if 30+ sentences). Standard precision (+/- 1.5 grades). Full lexical diversity scoring.
Document 600+ words	All formulas eligible including SMOG. Full lexical diversity. Highest confidence.
Corpus < 30 documents	Do not compute corpus-level aggregates (median, IQR, percentiles). Report per-document scores only.
Highly variable text lengths (CV > 1.0)	Raw TTR comparison across documents is INVALID. Use MTLD and MATTR exclusively. Report the CV and explain why TTR is excluded from cross-document comparison.
All metrics cluster in a narrow range	If the grade-level IQR is < 1.0 and MTLD IQR is < 10, the corpus is stylistically uniform. This is a finding, not a problem. Report the narrow range as a constraint: "complexity is tightly controlled at grade [X] +/- [Y]."
Bimodal distribution detected	If readability or MTLD histograms show two peaks, the corpus may contain two distinct complexity populations. Segment by the bimodal feature and report separately.

Common Mistakes

Mistake	Fix
Running only Flesch-Kincaid and calling it "readability analysis"	Run all five formulas. Report consensus median and spread.
Comparing TTR between a 200-word text and a 5,000-word text	Use MTLD or MATTR for cross-document comparison. TTR is only valid within equal-length samples.
Reporting "FK grade 12.3" as a precise target	Report as "FK grade 11-14" (range). Individual scores have +/- 1.5 precision.
Computing SMOG on a 5-sentence text	SMOG requires 30 sentences. Skip SMOG for short texts and use the other four formulas.
Using readability scores on a single paragraph	Single paragraphs (< 100 words) produce unstable scores. Concatenate or exclude.
Treating MTLD of 90 vs 92 as meaningfully different	MTLD differences < 10 are likely noise. Report as ranges.
Not verifying that MTLD is length-independent in this corpus	Compute the MTLD-length correlation. If r > 0.3, something is wrong (possible very short texts).
Producing metrics without translating to prompt constraints	The entire point is actionable constraints. Always end with a copy-paste constraint block.
Averaging all documents including 10-word ones	Enforce minimum thresholds. Exclude or concatenate sub-threshold texts.
Ignoring formula divergence	When FK and CLI disagree by 4+ grades, investigate why. The divergence is information about the text's structure.

References

• Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221-233.
• Kincaid, J.P. et al. (1975). Derivation of new readability formulas for Navy enlisted personnel. Research Branch Report 8-75, Naval Technical Training Command.
• Coleman, M. & Liau, T.L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2), 283-284.
• Gunning, R. (1952). The Technique of Clear Writing. McGraw-Hill.
• McLaughlin, G.H. (1969). SMOG grading: A new readability formula. Journal of Reading, 12(8), 639-646.
• McCarthy, P.M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study. Behavior Research Methods, 42(2), 381-392.
• Covington, M.A. & McFall, J.D. (2010). Cutting the Gordian knot: The moving-average type-token ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94-100.
• textstat Python library
• lexical_diversity Python library
• TAALED: Tool for the Automatic Assessment of Lexical Diversity
• Readability Formulas Comparison Guide
• Investigating minimum text lengths for lexical diversity indices (2020)