Check Metadata Typos

Check Metadata Typos

Check metadata files for spelling typos using comprehensive spell checking.

Scope Options

Ask the user which scope they want to check:

1. Current step only - Ask the user to specify the step path (e.g., etl/steps/data/garden/energy/2025-06-27/electricity_mix)
2. All ETL metadata - Check all active .meta.yml files in etl/steps/data/{garden,meadow,grapher}/ (automatically excludes ~3,570 archived steps)
3. Snapshot metadata - Check all snapshot .dvc files in snapshots/ (~7,915 files)
4. All metadata - Check both ETL steps and snapshot metadata files

Note: Archived steps and snapshots (defined in dag/archive/*.yml) are automatically excluded from checking as they are no longer actively maintained.

Implementation Strategy

0. Check codespell installation

IMPORTANT: Check if codespell is installed before attempting to use it. Since codespell is now a dev dependency in the project, it should already be installed, but verify first to avoid reinstalling unnecessarily.

# Check if codespell is installed
if ! .venv/bin/codespell --version &> /dev/null; then
    echo "codespell not found, installing..."
    uv add --dev codespell
else
    echo "codespell is already installed"
fi

If codespell is not installed and uv add --dev codespell fails, explain to the user how to install it manually.

1. Exclude archived steps and snapshots

IMPORTANT: Do not check archived steps and snapshots as they are no longer in use.

Archived steps and snapshots are defined in dag/archive/*.yml files:

• ~3,570 deprecated steps (garden, meadow, grapher)
• ~736 deprecated snapshots

To exclude them, extract their paths and create a list of active files:

# Extract archived step paths to a file
for step_type in garden meadow grapher; do
  grep -h "data://${step_type}/" dag/archive/*.yml 2>/dev/null | \
    grep -o "data://${step_type}/[^:]*" | \
    sed 's|data://|etl/steps/data/|' | \
    sed 's|$|.meta.yml|'
done > /tmp/archived_files.txt

# Extract archived snapshots
grep -rh "snapshot://" dag/archive/*.yml 2>/dev/null | \
  grep -o "snapshot://[^:]*" | \
  sed 's|snapshot://|snapshots/|' | \
  sed 's|$|.dvc|' | \
  sort -u >> /tmp/archived_files.txt

# Create list of all metadata files
find etl/steps/data/garden -name "*.meta.yml" > /tmp/all_meta_files.txt
find etl/steps/data/meadow -name "*.meta.yml" >> /tmp/all_meta_files.txt
find etl/steps/data/grapher -name "*.meta.yml" >> /tmp/all_meta_files.txt
find snapshots -name "*.dvc" >> /tmp/all_meta_files.txt

# Filter out archived files
grep -vFf /tmp/archived_files.txt /tmp/all_meta_files.txt > /tmp/active_meta_files.txt

echo "Total files to check: $(wc -l < /tmp/active_meta_files.txt)"

2. Run codespell with ignore list and exclusions

Use the existing .codespell-ignore.txt file to filter out domain-specific terms.

Whichever option the user picked, write that scope's file list to /tmp/codespell_targets.txt and check it. Steps 5 and 6 reuse that one file, so the fix and the verification cannot act on a wider scope than the check did. Never point a later step at a different list.

For option 1 (current step only):

Ask the user for the path, then normalize it. This scope has to accept three shapes, because /create-snapshot step 5 calls it on a single new snapshot file:

• an ETL step directory (etl/steps/data/garden/energy/2025-06-27/electricity_mix) → every .meta.yml in it
• a snapshot stem with no extension (snapshots/who/2026-04-22/mortality) → the matching .dvc
• a single file, .meta.yml or .dvc → itself

# For specific step or single file (option 1)
TARGET="<user_provided_path>"

.venv/bin/python - "$TARGET" > /tmp/codespell_targets.txt <<'PY'
import glob, os, sys

target = sys.argv[1].rstrip("/")
if os.path.isdir(target):
    paths = sorted(glob.glob(f"{target}/*.meta.yml")) + sorted(glob.glob(f"{target}/*.dvc"))
elif os.path.isfile(target):
    paths = [target]
else:
    # A snapshot stem: snapshots/<namespace>/<version>/<short_name>, extension unknown.
    paths = sorted(glob.glob(f"{target}.*.dvc")) or sorted(glob.glob(f"{target}.dvc"))
print("\n".join(paths))
PY

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Two ways this scope silently checks nothing, both of which must be treated as an error rather than a clean result:

• A .dvc target matched by a *.meta.yml-only pattern. A snapshot path has no .meta.yml in it, so a .meta.yml-only search returns an empty list and codespell reports no typos on a file it never opened. The normalizer above exists for exactly this case.
• An empty list from a shell glob. Don't build the list with ls <path>/*.meta.yml: under zsh a glob matching nothing aborts the command before the redirect, so no file is written and a wider list left over from an earlier run gets reused.

Always report the number of files in /tmp/codespell_targets.txt before running codespell. If it is 0, say the path matched nothing and ask for a corrected one — never fall through to a wider scope, and never report "no typos found".

For option 2 (all ETL metadata - garden, meadow, grapher):

# For all ETL step metadata (option 2)
find etl/steps/data/garden -name "*.meta.yml" > /tmp/all_step_files.txt
find etl/steps/data/meadow -name "*.meta.yml" >> /tmp/all_step_files.txt
find etl/steps/data/grapher -name "*.meta.yml" >> /tmp/all_step_files.txt
grep -vFf /tmp/archived_files.txt /tmp/all_step_files.txt > /tmp/codespell_targets.txt

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Note: Excluding archived steps reduces the scope by ~3,570 files and focuses on actively maintained metadata.

For option 3 (snapshot metadata):

# For all snapshot metadata (option 3)
find snapshots -name "*.dvc" > /tmp/all_snapshot_files.txt
grep -vFf /tmp/archived_files.txt /tmp/all_snapshot_files.txt > /tmp/codespell_targets.txt

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Note: the prose worth checking in a snapshot .dvc lives under meta.origin — description, description_snapshot, title, title_snapshot, citation_full, attribution. Older files instead use the deprecated meta.source.description / meta.source.published_by; both shapes are still in the repo (~2,000 files on origin, ~6,000 on source). codespell reads the whole file either way, so no filtering is needed — but report hits by their real field path, and don't "fix" a typo by migrating a file from source to origin (that's a separate change, out of scope here). ~736 archived snapshots are excluded.

For option 4 (all metadata):

# For all metadata - ETL and snapshots (option 4)
# Use the active_meta_files.txt created in step 1
cp /tmp/active_meta_files.txt /tmp/codespell_targets.txt

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

3. Parse and present results

Extract typos from codespell output and present them in a structured format:

• Group by typo type (e.g., all instances of "seperate" → "separate")
• Show file paths (as clickable links when possible)
• Show line numbers
• Show suggested corrections

Example output format:

Found 15 typos across 8 files:

Most common:
- "inmigrant" → "immigrant" (5 occurrences in 2 files)
- "seperate" → "separate" (3 occurrences in 1 file)
- "accomodation" → "accommodation" (2 occurrences in 1 file)

Detailed list:
[file.meta.yml:123] inmigrant → immigrant
[file.meta.yml:456] seperate → separate
...

4. Offer to fix typos

After presenting results, ask the user:

• Fix all automatically? - Apply all suggested fixes
• Review each typo? - Go through typos one by one for confirmation
• Cancel - Exit without making changes

5. Apply fixes (if user confirms)

For automatic fixes, let codespell apply its own corrections — it rewrites only the words it flagged, at the positions it flagged them. Run it over /tmp/codespell_targets.txt, the list step 2 wrote for the scope the user chose:

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt --write-changes

Never widen the scope here. --write-changes rewrites every file it is given, so pointing this at /tmp/active_meta_files.txt (the option-4 list) after the user asked for one step would silently edit thousands of unrelated files. The list must be the one that produced the reported typos — if /tmp/codespell_targets.txt is missing or you are unsure it matches, re-run step 2 for the chosen option before fixing anything.

For reviewed fixes, apply each one with the Edit tool, matching enough surrounding words to be unambiguous.

Never fix with an unanchored sed substitution. sed -i 's/word/word/g' file rewrites every occurrence in the file, including the ones you decided to ignore — inside URLs, domain names, and ALL-CAPS acronyms (see Notes). It also silently rewrites a word codespell never flagged on that line. sed -i '' is BSD-only on top of that, so it breaks on Linux (CI, cloud sandbox) where the same command needs sed -i.

6. Verify fixes

After applying fixes, re-run codespell over the same scope to verify all typos were corrected:

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Should return 0 results. If the user chose to review typos one by one and deliberately skipped some, expect exactly those to remain — say which, rather than reporting the check as failed.

7. Clean up

IMPORTANT: Delete any temporary files created during the check:

rm -f /tmp/archived_files.txt /tmp/all_meta_files.txt /tmp/active_meta_files.txt \
      /tmp/all_step_files.txt /tmp/active_step_files.txt \
      /tmp/all_snapshot_files.txt /tmp/active_snapshot_files.txt \
      /tmp/codespell_output.txt

The only persistent files should be:

- The `.codespell-ignore.txt` whitelist (if it doesn't exist, create it)
- Modified `.meta.yml` files (if fixes were applied)

**Do NOT create new persistent files in the repo like:**

- ❌ `TYPO_CHECK_REPORT.md`
- ❌ `scripts/analyze_typos.py`
- ❌ `scripts/advanced_spell_checker.py`

All analysis logic should be embedded in this command execution, not saved as separate files.

---

## Error Handling

- Check if codespell is installed first (see step 0). If not installed and `uv add --dev codespell` fails, explain to the user how to install it manually with `uv sync` or check their Python environment
- If no `.meta.yml` or `.dvc` files are found in the specified scope, inform the user
- If codespell finds no typos, congratulate the user on clean metadata!
- If file modification fails, report which files couldn't be updated

---

## Notes

- Always use American English spelling (e.g., "combating" not "combatting")
- Technical field names (like variable names with underscores) are typically safe to ignore
- **Acronyms in ALL CAPS should be ignored** - they are almost always legitimate acronyms (e.g., TE, INE, DIEA)
- **URLs and domain names should be ignored** - codespell may flag parts of URLs (e.g., "ine.es", "corona.fo") but these are correct
- When in doubt about a flagged word, ask the user before fixing