Readability Lexical Diversity

Readability and Lexical Diversity Metrics

Overview

Compute quantitative complexity targets from a text corpus by measuring readability (grade-level estimates from multiple formulas) and lexical diversity (vocabulary richness corrected for text length). The core principle: no single readability formula or diversity index is sufficient -- use multiple measures, report the range, and correct for text length before comparing. Readability formulas estimate the US grade level required to comprehend text; lexical diversity indices estimate how varied the vocabulary is. Both translate directly into measurable LLM prompt constraints (e.g., "target FK grade 12-14, MTLD above 80, average sentence length 18-22 words").

Why multiple formulas: Flesch-Kincaid and Coleman-Liau can diverge by 2-4 grade levels on the same text because they weight different proxies for complexity (syllables vs. characters, sentence length vs. word length). Reporting only one formula hides this disagreement. The consensus range across formulas is the reliable signal.

When to Use

• Measuring the complexity profile of a text corpus before voice replication
• Establishing quantitative readability targets for LLM prompt constraints
• Comparing complexity across documents, authors, or time periods within a corpus
• Computing lexical diversity with length-corrected measures (MTLD, MATTR)
• Setting a complexity ceiling and floor for generated text
• Feeding readability/diversity metrics into an archetype or persona construction pipeline

When NOT to use:

• Texts shorter than 100 words (readability formulas are unreliable; see Insufficient Data Handling)
• Comparing readability scores across languages without recalibration (formulas are English-calibrated)
• Equating readability with writing quality (low grade level is not "bad"; high grade level is not "good")
• Claiming readability scores measure actual reader comprehension (they estimate surface complexity)
• Using raw TTR on variable-length texts without correction (TTR decreases mechanically as text length increases)

digraph readability_decision {
    rankdir=TB;
    "Texts 100+ words?" [shape=diamond];
    "English text?" [shape=diamond];
    "Variable text lengths\nin corpus?" [shape=diamond];
    "Full analysis:\nmultiple formulas +\nlength-corrected LD" [shape=box];
    "Report insufficient\ndata per formula" [shape=box];
    "Flag: formulas not\ncalibrated for this language" [shape=box];
    "Use MTLD/MATTR\nnot raw TTR" [shape=box];

    "Texts 100+ words?" -> "English text?" [label="yes"];
    "Texts 100+ words?" -> "Report insufficient\ndata per formula" [label="no"];
    "English text?" -> "Variable text lengths\nin corpus?" [label="yes"];
    "English text?" -> "Flag: formulas not\ncalibrated for this language" [label="no"];
    "Variable text lengths\nin corpus?" -> "Use MTLD/MATTR\nnot raw TTR" [label="yes, variable"];
    "Variable text lengths\nin corpus?" -> "Full analysis:\nmultiple formulas +\nlength-corrected LD" [label="no, uniform"];
}

Quick Reference

Readability Formulas

Lexical Diversity Indices

Grade Level Interpretation

Minimum Text Length Per Formula

Workflow

Copy this checklist and track progress:

Readability & Lexical Diversity Analysis Progress:
- [ ] Step 1: Validate corpus suitability (text lengths, language, count)
- [ ] Step 2: Compute per-document readability scores (all five formulas)
- [ ] Step 3: Compute per-document lexical diversity (MTLD, MATTR, hapax)
- [ ] Step 4: Compute corpus-level aggregate statistics
- [ ] Step 5: Analyze formula agreement and divergence
- [ ] Step 6: Translate metrics into LLM prompt constraints
- [ ] Step 7: Write findings to docs/analysis/19-readability-lexical-diversity.md

Step 1: Validate Corpus Suitability

Before computing metrics, assess every document against minimum thresholds.

Suitability checks:

import textstat
import re
from collections import Counter

def assess_document_suitability(text):
    """Assess whether a document meets minimum thresholds for each formula."""
    words = text.split()
    sentences = [s.strip() for s in re.split(r'[.!?]+', text) if s.strip()]
    word_count = len(words)
    sentence_count = len(sentences)

    return {
        'word_count': word_count,
        'sentence_count': sentence_count,
        'fk_eligible': word_count >= 100,
        'cli_eligible': word_count >= 100,
        'fog_eligible': word_count >= 100,
        'smog_eligible': sentence_count >= 30,
        'ari_eligible': word_count >= 100,
        'mtld_eligible': word_count >= 100,
        'mattr_eligible': word_count >= 50,
    }

If most documents fail suitability: Consider concatenating short documents into larger units (by author, by thread, by time window) before scoring. Document the concatenation strategy and note that scores reflect the aggregate unit, not individual documents.

Step 2: Compute Per-Document Readability Scores

Score every eligible document with all five formulas. Report all five -- never choose one formula and discard the rest.

import textstat
import numpy as np

def compute_readability(text, suitability):
    """Compute all readability scores for an eligible document.
    Returns None for formulas where the document is below minimum."""
    scores = {}

    if suitability['fk_eligible']:
        scores['flesch_kincaid'] = textstat.flesch_kincaid_grade(text)
    if suitability['cli_eligible']:
        scores['coleman_liau'] = textstat.coleman_liau_index(text)
    if suitability['fog_eligible']:
        scores['gunning_fog'] = textstat.gunning_fog(text)
    if suitability['smog_eligible']:
        scores['smog'] = textstat.smog_index(text)
    if suitability['ari_eligible']:
        scores['ari'] = textstat.automated_readability_index(text)

    # Structural features (always computable if 1+ sentence)
    words = text.split()
    sentences = [s.strip() for s in re.split(r'[.!?]+', text) if s.strip()]
    scores['avg_sentence_length'] = len(words) / max(len(sentences), 1)
    scores['avg_word_length_chars'] = np.mean([len(w) for w in words])
    scores['syllables_per_word'] = textstat.avg_syllables_per_word(text)

    # Consensus grade level (median of available formula scores)
    grade_scores = [v for k, v in scores.items()
                    if k in ('flesch_kincaid', 'coleman_liau', 'gunning_fog',
                             'smog', 'ari') and v is not None]
    if grade_scores:
        scores['consensus_grade'] = np.median(grade_scores)
        scores['grade_range'] = (min(grade_scores), max(grade_scores))
        scores['grade_spread'] = max(grade_scores) - min(grade_scores)

    return scores

Critical: The consensus_grade (median across formulas) and grade_range (min-max spread) are the primary outputs. Individual formula scores are evidence; the range is the finding.

Step 3: Compute Per-Document Lexical Diversity

Use length-corrected measures (MTLD, MATTR) as primary indices. Report raw TTR only as a supplementary reference, never as a standalone comparison across different-length texts.

from collections import Counter

def compute_lexical_diversity(text, suitability):
    """Compute lexical diversity with length correction.
    MTLD and MATTR are primary; TTR and hapax are supplementary."""
    words = text.lower().split()
    word_count = len(words)
    scores = {'token_count': word_count}

    if word_count == 0:
        return scores

    # Raw TTR -- supplementary only, do NOT compare across different lengths
    unique_words = set(words)
    scores['ttr_raw'] = len(unique_words) / word_count
    scores['type_count'] = len(unique_words)

    # Hapax legomena (words appearing exactly once)
    freq = Counter(words)
    hapax_count = sum(1 for w, c in freq.items() if c == 1)
    scores['hapax_count'] = hapax_count
    scores['hapax_ratio'] = hapax_count / word_count

    # MTLD -- primary length-corrected measure
    if suitability['mtld_eligible']:
        try:
            from lexical_diversity import lex_div as ld
            scores['mtld'] = ld.mtld(words)
        except ImportError:
            scores['mtld'] = _compute_mtld_fallback(words)

    # MATTR -- moving-average TTR (window=50 tokens)
    if suitability['mattr_eligible']:
        scores['mattr_50'] = _compute_mattr(words, window=50)

    return scores

def _compute_mattr(tokens, window=50):
    """Moving-Average Type-Token Ratio.
    Calculates TTR for each sliding window and averages."""
    if len(tokens) < window:
        return len(set(tokens)) / len(tokens)  # fall back to TTR
    ttrs = []
    for i in range(len(tokens) - window + 1):
        window_tokens = tokens[i:i + window]
        ttrs.append(len(set(window_tokens)) / window)
    return np.mean(ttrs)

def _compute_mtld_fallback(tokens, threshold=0.72):
    """MTLD: average segment length before TTR drops below threshold.
    Runs forward and backward, averages both passes."""
    def _one_pass(tokens):
        factor_count = 0
        factor_length = 0
        current_start = 0
        for i in range(1, len(tokens) + 1):
            segment = tokens[current_start:i]
            ttr = len(set(segment)) / len(segment)
            if ttr <= threshold:
                factor_count += 1
                factor_length += len(segment)
                current_start = i
        # Handle partial factor at end
        if current_start < len(tokens):
            remaining = tokens[current_start:]
            remaining_ttr = len(set(remaining)) / len(remaining)
            if remaining_ttr < 1.0:
                partial = (1.0 - remaining_ttr) / (1.0 - threshold)
                factor_count += partial
                factor_length += len(remaining)
        if factor_count == 0:
            return len(tokens)  # TTR never dropped below threshold
        return factor_length / factor_count

    forward = _one_pass(tokens)
    backward = _one_pass(list(reversed(tokens)))
    return (forward + backward) / 2

Why MTLD is primary: McCarthy and Jarvis (2010) found MTLD has negligible correlation with text length (r = -0.02), while TTR correlates strongly (r = -0.70 to -0.90). MTLD reflects how many consecutive words a writer sustains before repeating vocabulary. Higher MTLD = more diverse vocabulary. Typical ranges: 50-80 (moderate), 80-120 (high), 120+ (very high).

Step 4: Compute Corpus-Level Aggregate Statistics

Aggregate per-document scores into corpus-level distributions.

import pandas as pd

def compute_corpus_aggregates(doc_scores_df):
    """Compute corpus-level statistics from per-document scores."""
    metrics = ['flesch_kincaid', 'coleman_liau', 'gunning_fog', 'smog',
               'ari', 'consensus_grade', 'avg_sentence_length',
               'mtld', 'mattr_50', 'ttr_raw', 'hapax_ratio']

    aggregates = {}
    for metric in metrics:
        if metric not in doc_scores_df.columns:
            continue
        col = doc_scores_df[metric].dropna()
        if len(col) < 2:
            continue
        aggregates[metric] = {
            'mean': col.mean(),
            'median': col.median(),
            'std': col.std(),
            'min': col.min(),
            'max': col.max(),
            'p25': col.quantile(0.25),
            'p75': col.quantile(0.75),
            'n_scored': len(col),
        }

    return aggregates

Step 5: Analyze Formula Agreement and Divergence

This step answers the critical question: do the formulas agree, and where do they disagree?

Agreement analysis:

1. For each document, compute the spread (max grade - min grade across formulas)
2. Compute the corpus-level median spread
3. Flag documents where spread exceeds 4 grade levels -- these often contain unusual structures (very long sentences with simple words, or short sentences with technical jargon)

Interpreting divergence:

Step 6: Translate Metrics Into LLM Prompt Constraints

This is the critical output step: converting measured complexity into actionable constraints for text generation prompts.

Constraint mapping table:

Constraint output format (embed in prompts):

## Complexity Constraints
- **Grade level:** Target FK grade [X-Y] (consensus range from corpus analysis)
- **Sentence length:** Average [N-M] words per sentence; vary between [min] and [max]
- **Vocabulary diversity:** MTLD target [N]+; avoid repeating non-function words within [N] sentences
- **Word complexity:** Average [N] syllables per word; ~[N]% of words should be 3+ syllables
- **Lexical range:** Approximately [N]% of words should be unique (hapax target)

Translating ranges, not points: Always convert measured metrics into target ranges, not exact values. If measured FK = 12.3, the constraint should be "FK grade 11-14" (not "FK grade 12.3"). Readability scores have an inherent precision of approximately +/- 1.5 grade levels on texts of 200+ words.

Step 7: Write the Report

Write all findings to docs/analysis/19-readability-lexical-diversity.md.

Report Output Template

The final report MUST be written to docs/analysis/19-readability-lexical-diversity.md with this structure:

# Readability and Lexical Diversity Analysis

## Methodology
- **Corpus:** [N documents scored, N excluded for insufficient length, total word count]
- **Readability formulas:** Flesch-Kincaid, Coleman-Liau, Gunning Fog, SMOG, ARI (via textstat)
- **Lexical diversity:** MTLD, MATTR (window=50), raw TTR (supplementary), hapax legomena ratio
- **Minimum thresholds:** 100 words per document for readability, 30 sentences for SMOG, 100 tokens for MTLD
- **Aggregation:** Per-document scoring with corpus-level median, IQR, and range

## Corpus Suitability Assessment
- **Total documents:** [N]
- **Documents meeting 100-word minimum:** [N] ([%])
- **Documents meeting 30-sentence minimum (SMOG):** [N] ([%])
- **Language:** [English / mixed]
- **Text length distribution:** mean [N] words, median [N], std [N], range [min-max]
- **Text length CV:** [X.XX] [uniform / moderate / highly variable]
- **Overall suitability:** [suitable / suitable with caveats / unsuitable]

## Readability Scores

### Per-Formula Corpus Statistics
| Formula | Mean | Median | Std | P25 | P75 | Min | Max | N Scored |
|---------|------|--------|-----|-----|-----|-----|-----|----------|
| Flesch-Kincaid | | | | | | | | |
| Coleman-Liau | | | | | | | | |
| Gunning Fog | | | | | | | | |
| SMOG | | | | | | | | |
| ARI | | | | | | | | |

### Consensus Grade Level
- **Median consensus grade:** [X.X] (IQR: [X.X - X.X])
- **Corpus grade range:** [X - X] (min to max of per-document consensus)
- **Interpretation:** [audience level description]

### Formula Agreement Analysis
- **Median per-document spread:** [X.X] grade levels
- **Documents with spread > 4 grades:** [N] ([%])
- **Most divergent formula pair:** [Formula A] vs [Formula B] (mean difference: [X.X])
- **Interpretation:** [why the formulas disagree for this corpus]

### Structural Metrics
| Metric | Mean | Median | Std | P25 | P75 |
|--------|------|--------|-----|-----|-----|
| Avg sentence length (words) | | | | | |
| Avg word length (characters) | | | | | |
| Syllables per word | | | | | |

## Lexical Diversity

### Corpus Statistics
| Index | Mean | Median | Std | P25 | P75 | N Scored |
|-------|------|--------|-----|-----|-----|----------|
| MTLD | | | | | | |
| MATTR (w=50) | | | | | | |
| TTR (raw, supplementary) | | | | | | |
| Hapax ratio | | | | | | |

### Length Correction Verification
- **TTR-length correlation:** r = [X.XX] (should be strongly negative)
- **MTLD-length correlation:** r = [X.XX] (should be near zero)
- **MATTR-length correlation:** r = [X.XX] (should be near zero)
- **Conclusion:** [MTLD/MATTR confirmed length-independent / TTR confirmed length-dependent]

## LLM Prompt Constraints (Derived)

### Complexity Ceiling and Floor
| Metric | Floor (P25) | Target (Median) | Ceiling (P75) | Constraint |
|--------|-------------|-----------------|---------------|------------|
| Grade level | | | | "Write at FK grade [X-Y]" |
| Sentence length | | | | "Average [X-Y] words per sentence" |
| Vocabulary diversity (MTLD) | | | | "Maintain MTLD of [X]+" |
| MATTR | | | | "In any 50-word span, [X-Y]% unique words" |
| Word complexity | | | | "Average [X-Y] syllables per word" |

### Full Constraint Block (Copy-Paste Ready)
[Pre-formatted constraint block for direct embedding in LLM prompts]

## Limitations and Caveats
- Readability formulas estimate surface complexity, not comprehension difficulty
- Grade levels are approximate (+/- 1.5 grades on 200+ word texts)
- Different formulas weight different proxies; the consensus range is the reliable signal
- Lexical diversity scores reflect vocabulary range, not vocabulary quality or appropriateness
- SMOG requires 30+ sentences; documents below this threshold were excluded from SMOG scoring
- [Corpus-specific limitations from Step 1]
- Short texts (< 200 words) produce less stable scores; [N] documents were in this range
- These metrics do NOT measure: clarity, coherence, accuracy, persuasiveness, or engagement

## References
- Flesch, R. (1948). A new readability yardstick. *Journal of Applied Psychology*, 32(3), 221-233.
- Kincaid, J.P. et al. (1975). Derivation of new readability formulas for Navy enlisted personnel. (Research Branch Report 8-75). Naval Technical Training Command.
- Coleman, M. & Liau, T.L. (1975). A computer readability formula designed for machine scoring. *Journal of Applied Psychology*, 60(2), 283-284.
- Gunning, R. (1952). *The Technique of Clear Writing*. McGraw-Hill.
- McLaughlin, G.H. (1969). SMOG grading: A new readability formula. *Journal of Reading*, 12(8), 639-646.
- McCarthy, P.M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. *Behavior Research Methods*, 42(2), 381-392.
- Covington, M.A. & McFall, J.D. (2010). Cutting the Gordian knot: The moving-average type-token ratio (MATTR). *Journal of Quantitative Linguistics*, 17(2), 94-100.
- textstat Python library: https://github.com/textstat/textstat
- lexical_diversity Python library: https://github.com/kristopherkyle/lexical_diversity
- TAALED: https://github.com/LCR-ADS-Lab/TAALED

Good Patterns

• Use multiple readability formulas and report the range -- no single formula captures complexity; the consensus (median) and spread across FK, CLI, Fog, SMOG, and ARI is the reliable signal
• Correct TTR for text length using MTLD or MATTR -- raw TTR is mechanically coupled to text length and cannot be compared across documents of different sizes
• Compute per-document AND corpus-level metrics -- per-document scores show variance; corpus-level aggregates set the constraint targets
• Report grade levels as ranges, not points -- "FK 11-14" is honest; "FK 12.3" implies false precision
• Translate every metric into a concrete LLM prompt constraint -- the analysis is only useful if it produces a copy-paste-ready constraint block
• Verify length correction by computing TTR-length and MTLD-length correlations -- confirm that MTLD/MATTR are indeed length-independent for your corpus
• Establish both ceiling and floor -- complexity constraints need an upper and lower bound, not just a target
• Report the number of documents excluded and why -- transparency about data loss from minimum thresholds

Anti-Patterns

Boundaries

This skill SHOULD produce:

• Per-document readability scores from all five formulas with eligibility tracking
• Per-document lexical diversity scores (MTLD, MATTR, hapax ratio) with length correction
• Corpus-level aggregate statistics (median, IQR, range) for all metrics
• Formula agreement analysis (median spread, divergent pairs)
• Complexity ceiling and floor (P25 and P75 of consensus grade)
• A copy-paste-ready LLM prompt constraint block derived from the metrics
• Length-correction verification (TTR-length vs MTLD-length correlations)
• Written report at docs/analysis/19-readability-lexical-diversity.md

This skill should NOT:

• Treat grade levels as exact (always report ranges with +/- 1.5 grade precision)
• Compare readability across languages without noting that formulas are English-calibrated
• Claim readability scores measure actual reader comprehension (they estimate surface complexity)
• Use metrics on texts shorter than 100 words without flagging unreliability
• Present raw TTR as a valid cross-document comparison when text lengths differ
• Equate high readability scores with good writing or low scores with bad writing
• Use a single formula and discard the others
• Skip the constraint translation step (metrics without actionable constraints are incomplete)
• Produce a report without a corpus suitability assessment

Insufficient Data Handling

digraph insufficient_data {
    rankdir=TB;
    node [shape=diamond, style=filled, fillcolor="#f0f0f0"];

    start [label="Per-document\nword count?", shape=diamond];
    tiny [label="< 100 words", shape=box, fillcolor="#ffcccc"];
    short [label="100-199 words", shape=box, fillcolor="#fff3cc"];
    adequate [label="200-599 words", shape=box, fillcolor="#ffffcc"];
    good [label="600+ words", shape=box, fillcolor="#ccffcc"];

    start -> tiny [label="< 100"];
    start -> short [label="100-199"];
    start -> adequate [label="200-599"];
    start -> good [label="600+"];

    tiny_action [label="EXCLUDE from per-document scoring.\nContribute to corpus aggregate\nonly via concatenation.", shape=box, fillcolor="#ffcccc"];
    short_action [label="Score FK, CLI, Fog, ARI.\nSkip SMOG.\nFlag MTLD as low-confidence.\nWiden grade precision to +/- 2.5.", shape=box, fillcolor="#fff3cc"];
    adequate_action [label="Score FK, CLI, Fog, ARI.\nSMOG only if 30+ sentences.\nFull lexical diversity.\nStandard precision +/- 1.5.", shape=box, fillcolor="#ffffcc"];
    good_action [label="All formulas eligible\nincluding SMOG.\nFull lexical diversity.\nHighest confidence.", shape=box, fillcolor="#ccffcc"];

    tiny -> tiny_action;
    short -> short_action;
    adequate -> adequate_action;
    good -> good_action;
}

Common Mistakes

References

• Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221-233.
• Kincaid, J.P. et al. (1975). Derivation of new readability formulas for Navy enlisted personnel. Research Branch Report 8-75, Naval Technical Training Command.
• Coleman, M. & Liau, T.L. (1975). A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2), 283-284.
• Gunning, R. (1952). The Technique of Clear Writing. McGraw-Hill.
• McLaughlin, G.H. (1969). SMOG grading: A new readability formula. Journal of Reading, 12(8), 639-646.
• McCarthy, P.M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study. Behavior Research Methods, 42(2), 381-392.
• Covington, M.A. & McFall, J.D. (2010). Cutting the Gordian knot: The moving-average type-token ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94-100.
• textstat Python library
• lexical_diversity Python library
• TAALED: Tool for the Automatic Assessment of Lexical Diversity
• Readability Formulas Comparison Guide
• Investigating minimum text lengths for lexical diversity indices (2020)