Literate Programming

Literate Programming Skill

CRITICAL: This skill MUST be activated BEFORE making any changes to .nw files!

You are an expert in literate programming using the noweb system.

Reference Files

This skill includes detailed references in references/:

When to Use This Skill

Correct Workflow

1. User asks to modify a .nw file
2. YOU ACTIVATE THIS SKILL IMMEDIATELY
3. You plan the changes with literate programming principles
4. You make the changes following the principles
5. You regenerate code with make/notangle

Anti-pattern (NEVER do this)

1. User asks to modify a .nw file
2. You directly edit the .nw file ← WRONG
3. Later review finds literate quality problems
4. You have to redo everything

Remember

• .nw files are NOT regular source code files
• They combine documentation and code for human readers
• Literate quality is AS IMPORTANT as code correctness
• Bad literate quality = failed task, even if code works

Planning Changes

When making changes to a .nw file:

1. Read the existing file to understand structure and narrative
2. Plan with literate programming in mind:
   • What is the "why" behind this change?
   • How does this fit into the existing narrative?
   • What new chunks are needed? What are their meaningful names?
   • Where in the pedagogical order should this be explained?
3. Design documentation BEFORE writing code:
   • Write prose explaining the problem and solution
   • Use subsections to structure complex explanations
4. Decompose code into well-named chunks:
   • Each chunk = one coherent concept
   • Names describe purpose, not syntax (like pseudocode)
5. Write the code chunks
6. Regenerate and test

Key principle: If you find yourself writing code comments to explain logic, that explanation belongs in the documentation chunks instead.

Reviewing Literate Programs

When reviewing, evaluate:

1. Narrative flow: Coherent story? Pedagogical order?
2. Variation theory: Contrasts used? "Whole, parts, whole" structure?
3. Chunk quality: Meaningful names? Focused on single concepts?
4. Explanation quality: Explains "why" not just "what"?
5. Test organization: Tests after implementation, not before?
6. Proper noweb syntax: [[code]] notation? Valid chunk references?

Core Philosophy

Literate programming (Knuth) has two goals:

1. Explain to human beings what we want a computer to do
2. Present concepts in order best for human understanding (psychological order, not compiler order)

Variation Theory

Apply variation-theory skill when structuring explanations:

• Contrast: Show what something IS vs what it is NOT
• Separation: Start with whole (module outline), then parts (chunks)
• Generalization: Show pattern across different contexts
• Fusion: Integrate parts back into coherent whole

CRITICAL: Show concrete examples FIRST, then state general principles. Readers cannot discern a pattern without first experiencing variation.

Noweb File Format

Documentation Chunks

• Begin with @ followed by space or newline
• Contain explanatory text (LaTeX, Markdown, etc.)
• Copied verbatim by noweave

Code Chunks

• Begin with <<chunk name>>= on a line by itself (column 1)
• End when another chunk begins or at end of file
• Reference other chunks using <<chunk name>>
• Multiple chunks with same name are concatenated

Syntax Rules

• Quote code in documentation using [[code]] (escapes LaTeX special chars)
• Escape: @<< for literal <<, @@ in column 1 for literal @

Writing Guidelines

1. Start with the human story - problem, approach, design decisions
2. Introduce concepts in pedagogical order - not compiler order
3. Use meaningful chunk names - 2-5 word summary of purpose (like pseudocode)
4. Decompose by concept, not syntax
5. Explain the "why" - don't just describe what the code does
6. Keep chunks focused - single, coherent idea each
7. Use bucket chunks - accumulate <<functions>>= and <<constants>>= throughout
8. Define constants for magic numbers - never hardcode values
9. Co-locate dependencies with features - feature's imports in feature's section
10. Keep lines under 80 characters - both prose and code

LaTeX Documentation Quality

Apply latex-writing skill. Most common anti-patterns in .nw files:

Lists with bold labels: Use \begin{description} with \item[Label], NOT \begin{itemize} with \item \textbf{Label}:

Code with manual escaping: Use [[code]], NOT \texttt{...\_...}

Manual quotes: Use \enquote{...}, NOT "..." or ...'' ``

Manual cross-references: Use \cref{...}, NOT Section~\ref{...}

Progressive Disclosure Pattern

When introducing high-level structure, use abstract placeholder chunks that defer specifics:

def cli_show(user_regex,
             <<options for filtering>>):
  <<implementation>>
@

[... later, explain each option ...]

\paragraph{The --all option}
<<options for filtering>>=
all: Annotated[bool, all_opt] = False,
@

Benefits: readable high-level structure, pedagogical ordering, maintainability.

Chunk Concatenation Patterns

Use multiple definitions when building up a parameter list pedagogically:

\subsection{Adding the diff flag}
<<args for diff>>=
diff=args.diff,
@

[... later ...]

\subsection{Fine-tuning thresholds}
<<args for diff>>=
threshold=args.threshold
@

Use separate chunks when contexts differ (different scopes):

<<args from command line>>=  # Has args object
diff=args.diff,
@

<<params for recursion>>=    # No args, only parameters
diff=diff,
@

Test Organization

CRITICAL: Tests appear AFTER implementation, not before.

See references/testing-patterns.md for detailed patterns.

Key points:

• Distribute tests throughout file, near implementations
• Use single <<test functions>> chunk, concatenated
• Use from module import * in test file
• Frame tests pedagogically: "Let's verify this works..."

Multi-Directory Projects

For large projects (5+ .nw files), see references/multi-directory-projects.md.

Key structure:

project/
├── src/           # .nw files → .py + .tex
├── doc/           # Master document, preamble.tex
├── tests/         # Extracted test files
└── makefiles/     # Shared build rules (noweb.mk, subdir.mk)

LaTeX-Safe Chunk Names

Use [[...]] notation for Python chunks with underscores:

<<[[module_name.py]]>>=
def my_function():
    pass
@

Extract with: notangle -R"[[module_name.py]]" file.nw > module_name.py

Best Practices Summary

1. Write documentation first - then add code
2. Keep lines under 80 characters
3. Check for unused chunks - run noroots to find typos
4. Keep tangled code in .gitignore - .nw is source of truth
5. NEVER commit generated files - .py and .tex from .nw are build artifacts
6. Test your tangles - ensure extracted code runs
7. Keep docstrings independent from LaTeX - no \cref in docstrings
8. Include table of contents - add \tableofcontents in documentation

Git Workflow

See references/git-workflow.md for details.

Core rules:

• Only commit .nw files to git
• Add generated files to .gitignore immediately
• Regenerate code with make after checkout/pull
• Never commit generated .py or .tex files

Noweb Commands Quick Reference

See references/noweb-commands.md for details.

# Tangling
notangle -R"[[module.py]]" file.nw > module.py
noroots file.nw                              # List root chunks

# Weaving
noweave -n -delay -x -t2 file.nw > file.tex  # For inclusion
noweave -latex -x file.nw > file.tex         # Standalone

When Literate Programming Is Valuable

• Complex algorithms requiring detailed explanation
• Educational code where understanding is paramount
• Code maintained by others
• Programs where design decisions need documentation
• Projects combining multiple languages/tools