Repo Architecture Analysis

Repo Architecture Analysis & Module Maturity Classification

Scan Python repositories to discover packages, count classes/functions, classify maturity levels, and generate markdown/JSON reports.

When to Use

• User asks "what packages does this repo have?"
• User wants to know which modules lack tests
• User asks for a module maturity or readiness matrix
• User wants an architecture overview with diagrams
• Before prioritizing test coverage work

Three Scripts (workspace-hub)

1. scripts/analysis/repo_architecture_scanner.py — Basic package discovery, counting, Mermaid structure diagrams
2. scripts/analysis/architecture-scanner.py — Enhanced scanner with LOC, public API surface, import dependency graph, YAML output (#1604)
3. scripts/analysis/module_status_matrix.py — Maturity classification (PRODUCTION/DEVELOPMENT/SKELETON/GAP) (#1567)

Running

# Enhanced architecture scan with API surface + dependency graph (preferred)
uv run python scripts/analysis/architecture-scanner.py [REPO_PATH] [-o OUTPUT_DIR]
# Outputs: docs/architecture/api-surface-map.md + .yaml

# With collision detection (added #1627)
uv run python scripts/analysis/architecture-scanner.py --detect-collisions
# Appends collision report to api-surface-map.md, prints top-20 to stdout

# Basic architecture scan
uv run scripts/analysis/repo_architecture_scanner.py [REPO_PATH] [-o OUTPUT_DIR]

# Module status matrix (with LOC-weighted scoring + trend tracking, #1629)
uv run scripts/analysis/module_status_matrix.py [REPO_PATH] [-o OUTPUT_DIR]
# Defaults to docs/reports/ — use `-o docs/architecture` for canonical location
# Outputs: module-status-matrix.md + .json (JSON includes quality_score for trend tracking)

# All default to digitalmodel/ if no path given

Enhanced Scanner Features (architecture-scanner.py)

The enhanced scanner adds capabilities beyond the basic one:

• LOC counts per package (non-blank lines)
• Public API surface detection: classes/functions not prefixed with _
• Import dependency graph: scans for inter-package imports (absolute via namespace + relative)
• YAML report output alongside markdown
• Mermaid dependency graph (edges between packages, not just tree structure)
• Cross-package API name collision detection (--detect-collisions flag, #1627): builds reverse index of symbol→packages, flags symbols exported by 2+ packages (179 collisions found across 31 packages)
• Functions: discover_packages(), build_dependency_graph(), build_reverse_index(), detect_collisions(), generate_collision_report(), generate_yaml_report(), generate_markdown(), scan_repo()

Module Status Matrix Scoring (module_status_matrix.py)

Added in #1629 — LOC-weighted quality scoring and trend tracking:

• Quality score: test_count × file_count / LOC × 100 — higher = better tested per LOC
• Test-to-source ratio: test_count / file_count
• Trend tracking: compares current quality_score to previous JSON snapshot (↑↓→/NEW/—)
• Previous snapshot auto-loaded from existing module-status-matrix.json before overwrite
• Functions: compute_quality_score(), compute_test_source_ratio(), compute_trend()
• JSON output now includes loc, quality_score, test_source_ratio per package for future trend comparisons

Maturity Classification Rules

Priority order: GAP (only init.py) → PRODUCTION (>5 tests, >3 files, >50% docstrings) → DEVELOPMENT (≥1 test, not all files tiny) → SKELETON (everything else — 0 tests, or all files tiny).

Key Pitfalls (Discovered Through Iteration)

1. .egg-info directories break namespace detection

The src-layout scanner looks for a single child directory under src/ to detect namespace packages. .egg-info dirs count as children and break this heuristic.

Fix: Always filter out .egg-info and .dist-info:

subdirs = [d for d in src_dir.iterdir()
           if d.is_dir()
           and not d.name.startswith((".", "_"))
           and not d.name.endswith((".egg-info", ".dist-info"))]

2. .venv pollutes entry point discovery

rglob("__main__.py") will find hundreds of __main__.py files inside .venv/lib/python3.x/site-packages/. Same for scripts/ rglob hitting deep legacy trees.

Fix: Skip known directories:

skip_dirs = {".venv", "venv", "node_modules", ".git", "__pycache__", ".tox", ".nox"}
def _skip_venv(path):
    return any(part in skip_dirs for part in path.parts)

3. src-layout namespace detection

For repos using src/mypackage/ layout with a single namespace package, the scanner must detect this pattern and drill one level deeper to find the real sub-packages.

Pattern: If src/ has exactly 1 non-hidden, non-egg-info child with __init__.py, treat it as the namespace root and scan its children.

4. Entry point scripts/ scanning depth

Use glob("*.py") (top-level only) instead of rglob("*.py") for scripts/ to avoid listing hundreds of legacy/deep scripts.

5. Use relative paths in reports

Call path.relative_to(repo_path) for cleaner output instead of absolute paths.

TDD Structure

Tests for all scripts live in tests/analysis/:

• test_architecture_scanner.py — 19 tests (discovery, metrics, public API, dependency graph, YAML, markdown, e2e)
• test_collision_detection.py — 14 tests (reverse index, collision detection, dedup, report formatting, top-N, case sensitivity)
• test_repo_architecture_scanner.py — 17 tests (package discovery, counting, markdown, entry points)
• test_module_status_matrix.py — 18 tests (scanning, classification, docstring%, key classes, markdown, JSON)
• test_weighted_scoring.py — 13 tests (quality score, test/source ratio, trend comparison, edge cases)

All use tmp_path fixtures with mock repo structures to test each classification branch.

Hyphenated filenames need importlib (not sys.path.insert)

architecture-scanner.py has a hyphen, making it non-importable via normal Python import. Tests must use:

import importlib.util
_script_path = Path(__file__).resolve().parents[2] / "scripts" / "analysis" / "architecture-scanner.py"
_spec = importlib.util.spec_from_file_location("architecture_scanner", _script_path)
architecture_scanner = importlib.util.module_from_spec(_spec)
sys.modules["architecture_scanner"] = architecture_scanner
_spec.loader.exec_module(architecture_scanner)

Then test functions can from architecture_scanner import discover_packages normally.

6. Concurrent agent file conflicts

When running alongside other agents (Codex, Gemini), your files may get overwritten mid-session. Commit immediately after tests pass — don't leave uncommitted work sitting. If you hit index.lock, run rm -f .git/index.lock and retry.

7. scan_packages takes two arguments: src_dir and tests_dir

The actual working signature is scan_packages(src_dir: Path, tests_dir: Path) — this is more flexible than a single repo_path since it decouples source and test locations. The CLI flags --src-dir and --tests-dir map to these. The prior single-arg approach was less testable.

8. Test fixture for SKELETON vs GAP distinction

GAP = only __init__.py (no real source files). SKELETON = has source files but 0 tests. You do NOT need files >20 lines for DEVELOPMENT — the "all tiny" check only applies when combined with 0 tests. Keep fixture files small; the classification handles it correctly.

9. Packages without __init__.py are invisible — fix by adding init.py

If a directory lacks __init__.py, the scanner skips it. Fixed in #1626: data_models/ got __init__.py → scanner now reports 31 packages. When a package is missing, create an __init__.py with a module docstring. Commit goes to the digitalmodel repo (not workspace-hub — see pitfall #12).

10. Module status matrix summary format

The enhanced output includes a bold summary line at the top: **X/N PRODUCTION, Y/N DEVELOPMENT, Z/N SKELETON, W/N GAP**. This format was added to meet issue #1567 acceptance criteria.

12. digitalmodel/ is a separate git repo inside workspace-hub

digitalmodel/ is in workspace-hub's .gitignore — it's a separate repo (github.com/vamseeachanta/digitalmodel.git) cloned inside. Files under digitalmodel/src/ must be committed in the digitalmodel repo (cd digitalmodel && git add ... && git commit), while scripts/analysis/, tests/analysis/, docs/architecture/ are committed in workspace-hub. Before pushing either, stash unstaged changes from other terminals: git stash && git pull origin main --rebase && git stash pop && git push.

13. Module status matrix output path discrepancy

The script defaults to docs/reports/ but the canonical location used by other tools is docs/architecture/. Run with -o docs/architecture to update the canonical copy. Both locations exist and should be kept in sync.

11. Dependency graph only includes inter-package edges

The build_dependency_graph() function detects the namespace package name (e.g. digitalmodel) and only tracks imports between known sibling packages. Stdlib and third-party imports are excluded. It matches both absolute (from digitalmodel.pkg) and relative (from .pkg) import patterns.

Test Fixture Pattern for src-layout

@pytest.fixture
def sample_repo(tmp_path):
    src = tmp_path / "src" / "mypackage"
    src.mkdir(parents=True)
    (src / "__init__.py").write_text("")  # CRITICAL: must exist for namespace detection
    
    alpha = src / "alpha"
    alpha.mkdir()
    (alpha / "__init__.py").write_text('__all__ = ["Thing"]')
    (alpha / "engine.py").write_text('class Thing:\n    pass\n')
    
    tests_dir = tmp_path / "tests" / "alpha"
    tests_dir.mkdir(parents=True)
    (tests_dir / "test_engine.py").write_text("def test_thing(): pass\n")
    
    return tmp_path

Critical: The namespace package directory (src/mypackage/) MUST have __init__.py or it won't be detected as a namespace package.