Devmd Gap Analysis

DevMD Gap Analysis

Compares DevMD specification files against actual source code.

Arguments

/devmd-gap-analysis                          # code→doc: what did the docs miss?
/devmd-gap-analysis --reverse               # doc→code: what hasn't been implemented yet?
/devmd-gap-analysis SCHEMA.md                # Single file (default direction: code→doc)
/devmd-gap-analysis SCHEMA.md --reverse      # Single file, doc→code
/devmd-gap-analysis --source ../myapp        # Explicit source path
/devmd-gap-analysis --consistency-only       # Cross-file consistency check only (no source needed)

Two Directions

Code → Doc (default):
  Code has 23 tables → SCHEMA.md documents 18 → 5 gaps → "docs are incomplete"

Doc → Code (--reverse):
  SCHEMA.md defines 23 tables → code has 18 → 5 gaps → "implementation is incomplete"

When to Use (and When Not To)

Use (code→doc):

• DevMD files were written manually or edited after scan — verify they still match code
• Code changed since last scan — find what DevMD files are now stale
• Reviewing someone else's DevMD files against the actual project

Use (doc→code, --reverse):

• After /devmd-guide or manual spec writing — check how much has been built
• During implementation — track progress ("12/23 tables implemented")
• Before release — verify all specified features exist in code
• After /pdca do — confirm implementation matches the DevMD spec

Don't use:

• Immediately after /devmd-scan — that's self-grading. Scan has its own self-verification step.

Phase 0: Setup

0a. Language detection + recon

Same as /devmd-scan Phase 0. Reuse if run in same session.

0b. Inventory DevMD files

# Find all DevMD files
for f in PRODUCT GLOSSARY BRAND ARCHITECTURE SCHEMA API ERRORS LOGGING \
         DESIGN UI SCREENS FLOWS SEO TESTING SECURITY INFRA CONFIG DEVOPS \
         OPERATIONS CHANGELOG CLAUDE AGENTS HARNESS RUNTIME LIFECYCLE; do
  [ -f "${DEVMD_PATH}/${f}.md" ] && echo "PRESENT: ${f}.md ($(wc -l < "${DEVMD_PATH}/${f}.md") lines)"
done

Only analyze files that exist. Do not flag missing files as gaps — that's a tier decision, not a gap.

Phase 1: Deterministic Analysis (automated, reproducible)

For each DevMD file present, run the matching Count command from skills/common/patterns.md against the source code.

Direction determines what's a "gap"

Per-file counting

Code → Doc (default):

FILE: SCHEMA.md
  code_count: 23        # tables/models found in source (deterministic grep)
  doc_count: 18         # tables/models documented in SCHEMA.md
  coverage: 78%         # doc_count / code_count
  missing_from_doc:     # items in code but not in doc
    - audit_log         (src/models/audit-log.ts:5)
    - session           (src/models/session.ts:3)

Doc → Code (--reverse):

FILE: SCHEMA.md
  doc_count: 23         # tables/models defined in SCHEMA.md
  code_count: 18        # tables/models found in source (deterministic grep)
  implementation: 78%   # code_count / doc_count
  not_yet_implemented:  # items in doc but not in code
    - audit_log         (SCHEMA.md, frontmatter models.AuditLog)
    - analytics_event   (SCHEMA.md, frontmatter models.AnalyticsEvent)

Counting rules per file

Files NOT deterministically countable

These files describe qualitative concepts. Skip counting, handle in Phase 2.

• PRODUCT.md, GLOSSARY.md, BRAND.md, FLOWS.md, SCREENS.md, DESIGN.md
• LOGGING.md, SEO.md, HARNESS.md, RUNTIME.md, LIFECYCLE.md
• DEVOPS.md, OPERATIONS.md, CHANGELOG.md, CLAUDE.md, AGENTS.md

Phase 2: Evidence-Based Findings (LLM judgment, tagged)

For each DevMD file (including non-countable ones), an Agent reviews.

Agent prompt — Code → Doc (default)

You are auditing {FILE_NAME} against source code for accuracy and completeness.

## DevMD file content
{READ the DevMD file}

## Source files to check
{List of relevant source files from Phase 0 recon + patterns.md Glob results}

## Task

Find issues in these categories:

### A. Accuracy errors — claims in the doc that contradict source code
For each: quote the doc claim, quote the source code, explain the contradiction.

### B. Staleness — things in the doc that no longer exist in code
For each: quote the doc reference, show that Grep finds nothing.

### C. Coverage gaps (non-countable files only) — important source patterns not captured
For each: cite source file:line, explain what's missing and why it matters.

## Output format (strict)

Return ONLY a JSON array. No markdown, no explanation.

[
  {
    "type": "accuracy|staleness|coverage",
    "severity": "high|medium|low",
    "file": "{FILE_NAME}",
    "doc_claim": "exact quote from DevMD file or null",
    "source_file": "path/to/file.ts",
    "source_line": 42,
    "source_snippet": "1-3 lines of code",
    "description": "what's wrong, in one sentence"
  }
]

If no issues found, return [].
Do NOT invent issues. If you're unsure, don't include it.

Agent prompt — Doc → Code (--reverse)

You are checking whether {FILE_NAME} has been fully implemented in source code.

## DevMD file content
{READ the DevMD file}

## Source files to check
{List of relevant source files from Phase 0 recon + patterns.md Glob results}

## Task

For each item defined in the DevMD file, check if it exists in the source code.

### A. Not implemented — item defined in doc but not found in code
For each: quote the doc definition, show that Grep/Glob finds no matching implementation.

### B. Partially implemented — item exists but is incomplete
For each: quote the doc definition (full spec), quote the source code (partial), explain what's missing.

### C. Diverged — item is implemented but differently from the doc
For each: quote the doc spec, quote the source implementation, explain the difference.

## What counts as "defined in the doc"

- SCHEMA.md: every model/table and their fields
- API.md: every endpoint
- ARCHITECTURE.md: every package/layer
- ERRORS.md: every error code
- UI.md: every page/component
- FLOWS.md: every user journey
- SECURITY.md: every auth mechanism, every OWASP mitigation
- TESTING.md: every test type/suite mentioned
- CONFIG.md: every env var
- INFRA.md: every infrastructure component
- AGENTS.md: every agent/skill
- HARNESS.md: every LLM/RAG/tool config
- RUNTIME.md: every worker/cron/queue

## Output format (strict)

Return ONLY a JSON array. No markdown, no explanation.

[
  {
    "type": "not_implemented|partial|diverged",
    "severity": "high|medium|low",
    "file": "{FILE_NAME}",
    "doc_claim": "exact quote from DevMD file",
    "source_file": "path/to/file.ts or null if not found",
    "source_line": 42,
    "source_snippet": "1-3 lines of code or null",
    "description": "what's missing or different, in one sentence"
  }
]

Severity guide:
- high: core feature/table/endpoint not implemented at all
- medium: implemented but missing fields/params/error handling
- low: minor detail differs (naming, defaults)

If everything is implemented, return [].
Do NOT invent issues. If you're unsure, don't include it.

Confidence filter

After Agent returns, discard any finding where:

• source_file does not exist (verify with Glob)
• source_line does not contain source_snippet (verify with Read)
• doc_claim is not found in the DevMD file (verify with Grep)

This prevents hallucinated findings.

Phase 3: Cross-File Consistency

No source code needed. Compares DevMD files against each other.

3a. Cross-reference validation

# Extract all @FILE.md#section references
grep -rn "@[A-Z].*\.md" ${DEVMD_PATH}/*.md

For each reference:

1. Does the target file exist?
2. Does the target section (after #) exist as a heading in that file?

3b. Overlap contradiction detection

Check these specific pairs for semantic contradictions:

For each pair, an Agent reads both files and reports contradictions in the same JSON format as Phase 2 (type: "consistency").

Phase 4: Scoring

Deterministic score (from Phase 1)

deterministic_coverage = sum(doc_count) / sum(code_count) × 100

Only includes files that have countable items. Files with code_count = 0 are excluded.

Finding score (from Phase 2 + 3)

high_findings = count of severity=high
medium_findings = count of severity=medium
low_findings = count of severity=low

finding_penalty = (high × 5) + (medium × 2) + (low × 1)
finding_score = max(0, 100 - finding_penalty)

Consistency score (from Phase 3)

total_refs = count of @FILE.md references
broken_refs = count of references that don't resolve
contradictions = count of consistency findings

consistency_score = ((total_refs - broken_refs) / total_refs × 50) + (max(0, 50 - contradictions × 5))

Overall

overall = deterministic_coverage × 0.4 + finding_score × 0.35 + consistency_score × 0.25

Output

Write report to {devmd_path}/gap-analysis.md:

Code → Doc report (default)

# DevMD Gap Analysis — Code → Doc

**Direction**: Code → Doc (are docs complete?)
**Source**: {source_path}
**Date**: {ISO date}
**Overall Score**: {N}/100

## Scoring Breakdown

| Dimension | Score | Method | Details |
|-----------|-------|--------|---------|
| Coverage (deterministic) | {N}/100 | Grep counting | {doc_count}/{code_count} items across {N} files |
| Findings (evidence-based) | {N}/100 | LLM + verified | {high}H / {medium}M / {low}L findings |
| Consistency | {N}/100 | Cross-ref + overlap | {broken_refs} broken refs, {contradictions} contradictions |

## Deterministic Coverage

| File | In Code | In Doc | Coverage | Missing from Doc |
|------|---------|--------|----------|-----------------|
| SCHEMA.md | 23 | 18 | 78% | audit_log, session, ... |
| API.md | 45 | 42 | 93% | POST /webhooks, ... |

## Verified Findings
{findings grouped by severity with evidence}

## Consistency Issues
{cross-references + contradictions}

## Recommendations (ordered by impact)
1. ...

Doc → Code report (--reverse)

# DevMD Gap Analysis — Doc → Code

**Direction**: Doc → Code (is the spec implemented?)
**Source**: {source_path}
**Date**: {ISO date}
**Implementation Score**: {N}/100

## Implementation Progress

| File | Specified | Implemented | Progress | Not Yet Implemented |
|------|-----------|-------------|----------|---------------------|
| SCHEMA.md | 23 | 18 | 78% | audit_log, analytics_event, ... |
| API.md | 45 | 38 | 84% | DELETE /tokens, PATCH /billing, ... |

## Not Implemented (High Severity)

#### SCHEMA.md — audit_log table
- **Spec**: SCHEMA.md defines `AuditLog` with 8 fields (user_id, action, target_type, ...)
- **Code**: No file matching `*audit*model*` or `*audit*entity*` found
- **Impact**: Audit trail feature entirely missing

## Partially Implemented

#### API.md — PATCH /users/:id
- **Spec**: API.md defines 6 updatable fields (name, email, avatar, timezone, language, theme)
- **Code**: `src/routes/users.ts:45` handles only 3 fields (name, email, avatar)
- **Missing**: timezone, language, theme not in update handler

## Diverged from Spec

#### ERRORS.md — AUTH_003 code
- **Spec**: ERRORS.md defines AUTH_003 as "MFA_REQUIRED" (403)
- **Code**: `src/errors/auth.ts:12` defines AUTH_003 as "SESSION_EXPIRED" (401)

## Summary

- **Fully implemented**: {N} files at 100%
- **In progress**: {N} files with gaps
- **Not started**: {N} files with 0% implementation
- **Next priority**: {highest-impact unimplemented item}

Single-File Mode

When invoked as /devmd-gap-analysis SCHEMA.md:

1. Run Phase 0 (recon)
2. Run Phase 1 counting for SCHEMA.md only
3. Run Phase 2 Agent for SCHEMA.md only
4. Skip Phase 3 (cross-file consistency needs multiple files)
5. Score using Phase 1 + 2 only (no consistency dimension)
6. Output abbreviated report

Consistency-Only Mode

When invoked as /devmd-gap-analysis --consistency-only:

1. Skip Phase 0, 1, 2
2. Run Phase 3 only
3. Score consistency dimension only
4. Useful for checking DevMD files against each other without source code