Normalizing Text For Measurement

Normalizing Text for Measurement

Any pipeline that reduces a document to a number — word count, reading time, readability, a token budget, a dedup key, a checksum, a search index — has two halves: a normalizer that turns markup into something comparable, and a measurer that consumes it. Reviews concentrate on the measurer. Almost every bug in this shape lives in the normalizer, and none of them raise.

The failure mode is uniform: the output is still a plausible string, the metric is still a plausible number, and nothing anywhere reports a problem. See reporting-derived-metrics for the other end of the same pipeline — what the measurer does when the sample is too small.

A missing separator fuses two blocks into one token

A token-walking stripper emits text for content tokens and separators for close tokens. Miss one close token and two blocks become one word.

# Bad — paragraphs are separated, headings are not
for token in tokens:
    if token.type == "inline":
        out.append(token.content)
    elif token.type == "paragraph_close":
        out.append("\n\n")

"# Title\n\nBody text here." normalizes to "TitleBody text here.". One word instead of four, and a word that appears in no dictionary. Every consumer inherits it:

• word count drops by one per heading
• average word length rises, so readability scores report plain prose as far harder than it is — and the error scales with heading density, so the most structured documents are the most wrong
• a dedup key or checksum diverges from the same prose written without a heading

The general rule: every block-level construct the parser opens must close with a separator. Enumerate the close-token types your parser emits (heading_close, list_item_close, blockquote_close, table_row_close, …) and account for each one explicitly. A stripper that handles two of six is not "mostly right"; it is wrong on every document containing the other four.

Both metric classes need normalized text — they break on different residue

Every metric consumes normalized text. Classifying a metric does not tell you whether it needs the normalizer; it tells you which failure to hunt for first.

• Character-based metrics (reading time computed as chars × ms_per_char, byte-size limits, character quotas, rough token estimates) count every character of syntax as something a human reads. They are ruined by leftover syntax characters: a link's target URL, an image's path, fence delimiters, table pipes. On link- and image-heavy prose a character-based reading time can run roughly double the true figure; a single small table adds a noticeable double-digit percentage. The overshoot grows with markup density, so it is worst on exactly the documents (docs pages, READMEs, changelogs) the estimate matters most for.
• Word- and syllable-based metrics (word count, readability grade levels, syllable ratios) tokenize on whitespace and punctuation, so leftover delimiters largely wash out — table pipes leave readability scores essentially unchanged. What ruins them is a lost block boundary: the fused heading in the previous section corrupts the word and syllable stream itself, and no amount of downstream tokenizing recovers it.

Audit rule: for every metric, confirm it consumes normalized text. Then check the failure its class is blind to — character-based ones for syntax that survived the strip, word-based ones for boundaries the strip failed to insert. Both are verified the same way, by comparing against a plain-text equivalent rather than asserting a shape.

Enumerate every consumer of the normalizer

The most common version of this bug is not a broken normalizer — it is a sibling function that never calls it.

def readability(text: str) -> dict:
    return score(strip_markup(text))     # normalized

def reading_time(text: str) -> float:
    return textstat.reading_time(text)   # not normalized — same module, same input

Nothing fails. The two functions take the same argument, live in the same file, are described the same way in the docs, and disagree by a factor of two on any markdown input. Whenever you add or fix a normalizer:

1. Grep for every call site of the measurer, not the normalizer, and check each one goes through normalization.
2. Grep for every call site of the normalizer and note them — the count matters in both directions. Once a stripper has two production consumers, a one-line change to it moves two public outputs, and the PR description has to say so.
3. Check the segmented paths too. If the API exposes level="full" | "section" | "paragraph", a fix applied to the full-document path may leave the segmented ones on the old behaviour.

A parser preset is a rule set you did not choose

Constructing a parser with no arguments selects a default profile, and that profile decides which constructs are recognized at all.

md = MarkdownIt()          # CommonMark preset — no `table` rule

A GFM table under this preset is not a table. It is an ordinary paragraph whose text happens to contain | and ---, so every delimiter survives normalization into what you are calling "plain text". Nothing errors, because a paragraph is a perfectly valid parse.

Before trusting a parse tree, print the enabled rule set (md.get_active_rules() or the library's equivalent) and confirm the constructs you actually receive are in it. When you enable a rule, the stripper needs matching close-token handling in the same change — enabling table without emitting a space at cell close and a newline at row close just moves the problem.

Treat this as one case of a general habit: verify what a dependency actually does on your input rather than what its README implies. See verifying-external-behavior.

Grouping by a natural key that is not unique

Segmenting a document into sections and returning them as a dict keyed by heading text assumes headings are unique. They are not — ### Example, ## Notes, and ## Usage repeat all through real documents.

# Bad — the second "## Notes" overwrites or is discarded
sections = {heading_text: body for heading_text, body in walk(doc)}

Two consequences, and the second is the one people miss:

• Collapse. The first occurrence keeps the key; later ones are never reported under any key of their own. A document with four sections returns three.
• Double counting. The lost content usually is not dropped. A hierarchy stack extends it into the nearest ancestor section, so the ancestor's word count and reading time silently include a body that is also reported elsewhere. With no ancestor, it appears only in the full-document text. So the per-section numbers neither sum to the whole nor stay disjoint.

Key sections by something structurally unique — an index, a (level, index) pair, or a slug with a disambiguating suffix — and carry the display heading as a field. Then audit downstream: any dict comprehension over the sections and any processed_keys-style dedup guard was written assuming uniqueness too.

Write down which lossiness is deliberate

Normalizers are intentionally lossy in places. A stripper may drop list markers so - a\n- b measures as two bare items, or ignore raw HTML block content entirely. Those are design decisions, and they are indistinguishable at a glance from the missing-separator bug above.

Both directions of the mistake happen: a deliberate drop gets "fixed" by someone who read it as a defect, and an accidental fusion gets pinned by a test and survives for years as apparent intent. Put a one-line comment stating the intent next to the branch, and make the test name say it (test_list_markers_are_dropped, not test_list_output). A future reader should be able to tell which branches are decisions without reconstructing the argument.

Testing a normalizer without depending on a third-party constant

Two rules specific to this pipeline; for mutation-testing technique generally see testing-python-libraries.

Assert equivalence, not magic numbers. The strongest assertion needs no constant from the analysis library at all: the markdown input must measure identically to its hand-written plain-text equivalent.

def test_markdown_measures_as_its_plain_equivalent():
    markdown = "# Title\n\nSome [linked](https://example.com/a/long/path) text.\n"
    plain = "Title\n\nSome linked text.\n"
    assert reading_time(markdown) == reading_time(plain)
    assert strip_markup(markdown).strip() == "Title\n\nSome linked text."
    assert len(strip_markup(markdown).split()) == 3

This fails hard when the stripper regresses, and keeps passing when the scoring library is upgraded. If the analysis dependency is unpinned, an exact-float assertion on its output is version-brittle: the same document can score several points differently across releases. Where you want a number for documentation, use a tolerance sized against the bug's own gap (if the defect moved a score by ~30 points, abs=1.0 is both a real value assertion and version-tolerant), and measure the expected value by running the code — never estimate it by hand. These constants are not intuitive and a plausible guess fails CI.

A test that pins buggy output is worse than no test. When you find a normalizer defect, read its test file before assuming it is uncovered — often the wrong output is already asserted verbatim, which is why nobody noticed. Fixing the bug means updating those assertions, and there is usually more than one: the obvious unit test named after the construct, and a longer "mixed content" or "end to end" fixture whose expected string contains the same fused or dropped text mid-way. Scope the change by grepping the fixture bodies for the affected text, not by the test names.

A whitespace-only or empty input test guards nothing here. Most measurers already return 0 for " \n\n " before any fix, so such a test passes on the broken code and reads as coverage it is not.

Review checklist

• Every block-level close token the parser emits has an explicit separator branch; the list was enumerated from the parser, not from memory
• Every metric consumes normalized text; character-based ones additionally audited for leftover syntax characters, word-based ones for fused block boundaries
• Every call site of the measurer goes through the normalizer — including segmented (section/paragraph) paths, not just the full-document path
• Number of production consumers of the normalizer is known, and a change to it names every output it moves
• Parser rule set confirmed to include the constructs the input actually uses; newly enabled rules have matching close-token handling
• Section/segment maps keyed by something structurally unique, not by heading text; downstream comprehensions and dedup guards audited for the same assumption
• Deliberate lossiness commented at the branch and named in the test
• Regression tests assert markdown-equals-plain-equivalent plus the exact stripped string and word count, rather than a float from an unpinned library
• Existing tests checked for pinned buggy output before declaring the defect untested