Quarto

Quarto Skill

Quarto is an open-source scientific and technical publishing system built on Pandoc. It renders computational documents (with Python, R, Julia code) to publication-quality output in multiple formats.

Bootstrap with epq scaffold (PREFERRED for new projects)

New QMD analysis projects must be created via the epq CLI — do NOT manually create pyproject.toml, _quarto.yml, justfile, figures/_style.py, or latex-header.tex:

epq scaffold ~/workspace/projects/my-analysis   # generates all boilerplate
cd ~/workspace/projects/my-analysis
just bootstrap                                   # uv sync + ipykernel install
epq audit .                                      # verify clean (0 warnings)

epq scaffold generates: _quarto.yml (with jupyter: set), latex-header.tex (local copy — no external path dep at render time), justfile (thin import wrapper for canonical recipes), pyproject.toml (package = false, epq editable install), .gitignore, figures/fig_example.py (canonical dev loop), {name}_files/figure-pdf/ pre-created.

To audit or retrofit an existing project:

epq audit <path>    # JSON violations with file/line/suggestion
epq fix <path>      # unified diffs for auto-fixable issues (LLM reviews and applies)
epq list-rules      # enumerate all rule IDs

Shared library — import in every analysis QMD setup cell:

from epq import style, cache, bq, fmt

style.apply_style()          # canonical rcParams (150/200 DPI, sans-serif fallbacks)
style.NAVY, style.TEAL, ...  # workspace palette — never redefine inline
cache.read_cache("name")     # 24h TTL file-based cache → None if stale/missing
bq.run_bq_query(SQL)         # BigQuery client library wrapper → Iterator[DataFrame]
fmt.millions_formatter()     # FuncFormatter for ax.yaxis.set_major_formatter()

Full authoring reference: ~/src/analysis-doc/docs/AGENTS.md Retrofit guide: ~/src/analysis-doc/docs/RETROFIT.md

External Python Figures Pattern (PREFERRED for complex documents)

For documents with multiple visualizations, extract all matplotlib code into standalone Python modules in figures/. The QMD becomes a thin shell with stub cells only.

Architecture

{name}.qmd          ← thin shell: prose + data-load cells + stub figure cells only
_quarto.yml         ← jupyter: {name} (set by epq scaffold)
latex-header.tex    ← local copy (set by epq scaffold)
justfile            ← imports ~/src/analysis-doc/tools/justfile
figures/
  __init__.py
  fig_NAME.py       ← one module per figure; render(data) contract
scripts/data/
  extract_NAME.py   ← standalone BigQuery extractor; writes data/cache/*.json
data/cache/         ← JSON cache files (gitignored)
{name}_files/
  figure-pdf/       ← dev loop writes here (pre-created by epq scaffold)

Figure Module Contract

One file per figure (not a dispatcher). render(data: dict) is the only public function.

# figures/fig_revenue.py
from pathlib import Path
from epq import style, fmt  # palette and formatters from epq — never local _style.py

LABEL = "fig-revenue"       # matches QMD #| label: fig-revenue
FIG_WIDTH = 8.5
FIG_HEIGHT = 4.0
FIG_CAP = "Insight-focused caption — not a data description."


def render(data: dict) -> None:
    """Render figure. Called from QMD stub cell with shared data dict.

    Do NOT call plt.show() or plt.savefig() here — Quarto handles capture.
    Do NOT call plt.close() here — handled in __main__ and between QMD cells.
    """
    import matplotlib.pyplot as plt
    style.apply_style()
    fig, ax = plt.subplots(figsize=(FIG_WIDTH, FIG_HEIGHT))

    # ... all matplotlib code, reading from data dict ...
    df = data.get("revenue", _load_sample_data())
    ax.bar(df["month"], df["revenue"], color=style.NAVY)
    ax.yaxis.set_major_formatter(fmt.millions_formatter())
    ax.set_title(FIG_CAP.rstrip("."))
    plt.tight_layout()


def _load_sample_data():
    """Synthetic fallback for dev loop (no BQ needed)."""
    import pandas as pd
    return pd.DataFrame({"month": ["Q1", "Q2", "Q3", "Q4"],
                         "revenue": [1.2e6, 1.4e6, 1.3e6, 1.6e6]})


if __name__ == "__main__":
    """Dev loop — saves to {project}_files/figure-pdf/{LABEL}-output-1.png.

    Writes to the same path Quarto uses, so visual inspection is against the
    real render artifact. Run via: just dev-fig revenue
    """
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt

    _project_root = Path(__file__).parent.parent
    _out_dir = _project_root / f"{_project_root.name}_files" / "figure-pdf"
    _out_dir.mkdir(parents=True, exist_ok=True)
    out = str(_out_dir / f"{LABEL}-output-1.png")

    render({})                                          # synthetic data fallback
    plt.savefig(out, dpi=150, bbox_inches="tight")     # savefig BEFORE close
    print(f"Saved {out}")
    plt.close("all")

QMD Stub Cell

#| label: fig-revenue
#| fig-cap: "Revenue by stream."
#| fig-height: 4.0
#| fig-width: 8.5
#| fig-pos: "H"
#| out-width: 100%
import sys; sys.path.insert(0, str(Path("."))) if "." not in sys.path else None
from figures import fig_revenue
fig_revenue.render(data)

Justfile (generated by epq scaffold)

Projects get a thin justfile that imports the canonical recipe library:

# Project-local overrides go here. Canonical recipes imported below.
import? '~/src/analysis-doc/tools/justfile'

The canonical justfile provides:

# Render figure to {project}_files/figure-pdf/fig-NAME-output-1.png
dev-fig NAME:
    PYTHONPATH=. uv run python figures/fig_{{NAME}}.py
    @echo "→ $(basename $PWD)_files/figure-pdf/fig-{{NAME}}-output-1.png"

# Full render (clears Jupyter cache)
render:
    rm -rf .jupyter_cache/
    quarto render *.qmd

Figure Audit and Visual Iteration Protocol

CRITICAL: Any task involving figures — iteration, audit, or review — MUST render and visually inspect the PNG before reporting. Code-only review is incomplete.

Required sequence for every figure change or audit:

1. just dev-fig NAME → {project}_files/figure-pdf/fig-NAME-output-1.png
2. Read the PNG using the Read tool — do not skip this step
3. Apply the visual readability checklist to what you see in the PNG
4. Fix issues, re-render, re-inspect until checklist passes
5. Only then check factual accuracy against prose/data

Do NOT use just preview-fig / Playwright — it screenshots HTML chrome, not the raw figure. Read the PNG directly from {project}_files/figure-pdf/.

Visual Readability Checklist (inspect the rendered PNG, not the code):

• Suptitle not clipped — suptitle(y=1.02) ALWAYS clips in PDF. Use y=1.0 + fig.subplots_adjust(top=0.82–0.88). Never use y > 1.0.
• No text collisions — suptitle and panel titles have clear separation
• Text readable at print scale — PDF page is 6.5in wide; >1.5× zoom = illegible
• Panels not squished — 3-panel: FIG_HEIGHT ≥ 4.0; single-panel ≥ 3.0; swim-lane ≥ 4.5
• Bar labels within bounds — set_clip_on(True) makes labels invisible, not small
• No partial annotations — text near ylim/xlim edges fully in frame
• Text contrast correct — see color guard below

Text contrast rule (most common source of invisible text):

# Use epq.style helper — covers all palette fills correctly:
from epq import style
tc = style.text_color_for(fill_color)   # WHITE for dark fills, NAVY for light

# Manual guard if needed:
DARK_FILLS = (style.NAVY, style.TEAL, style.CORAL, style.PURPLE, style.GOLD)
tc = style.WHITE if fill in DARK_FILLS else style.NAVY
#                                               ^^^^ NAVY, never SLATE
# SLATE on LIGHT_SLATE = 2.6:1 contrast — fails AA, looks muddy at print scale

Verified contrast ratios:

• NAVY/TEAL/CORAL/PURPLE/GOLD fills → WHITE text (8–16:1 ✅)
• LIGHT_SLATE fill → NAVY text (5.5:1 ✅); SLATE fails (2.6:1 ❌)
• Pale backgrounds (LIGHT_BG, NAVY_BG) → NAVY text; WHITE is invisible (~1.07:1 ❌)

Common visual defects invisible in code:

• Suptitle bleeding into panel titles → increase FIG_HEIGHT, use subplots_adjust
• Bar labels extending past xlim → invisible (not small); expand xlim or reduce offset
• Blank/white PNG → plt.savefig() called after plt.close(). Fix: call savefig first:

  render({})
  plt.savefig(out, dpi=150, bbox_inches="tight")
  plt.close("all")
  

Key Rules

• Edit figures/fig_NAME.py, not the QMD — QMD stubs never change unless label/caption/dimensions change
• One module per figure — render(data: dict) -> None, no dispatcher pattern
• from epq import style, fmt — never a local _style.py copy
• PYTHONPATH=. required when running modules directly: PYTHONPATH=. uv run python figures/fig_NAME.py
• render() must NOT call plt.show(), plt.close(), or plt.savefig() — Quarto handles capture; __main__ handles save
• matplotlib.use("Agg") before any pyplot import in __main__
• data dict is the only input — never read cache files inside figure modules; provide synthetic fallback in _load_sample_data()
• Dev loop output: {project}_files/figure-pdf/{LABEL}-output-1.png (pre-created by epq scaffold)
• Reference implementation: ~/workspace/projects/luma-revenue-forecast/ and ~/workspace/projects/revenue-forecast-2026/

PDF Project Bootstrap

Use epq scaffold — it handles all of the following automatically:

epq scaffold ~/workspace/projects/my-project
cd ~/workspace/projects/my-project
just bootstrap    # runs: uv sync + ipykernel install + mkdir data/cache

Manual checklist (only if NOT using epq scaffold):

1. Create pyproject.toml with [tool.uv] package = false and run uv sync
2. Register kernel: uv run python -m ipykernel install --user --name=<project>
3. _quarto.yml: set jupyter: <project> (must match registered kernel name exactly)
4. Copy latex-header.tex locally from ~/src/analysis-doc/templates/ (do not reference it via external path)
5. Never borrow another project's venv or kernel — each project must own its own

PDF Title Suppression

• Never use YAML title: or subtitle: — they produce \maketitle which double-renders with any custom header
• Use a raw LaTeX inline header in the document body instead:

{\large\textbf{Document Title}}
\hfill
{\small\color{gray} Author \quad\textbullet\quad \today}
\vspace{4pt}
\hrule
\vspace{8pt}

• Add pagetitle: " " to YAML to suppress the HTML <title> without breaking Pandoc

PDF Figure Rules

Split multi-story panels

If a combined figure has two sub-panels telling different insights, split into separate fig-* cells with their own captions. Ask upfront whether panels should be split — this is cheaper than rework after the fact.

Legend placement

• Top-right (loc='upper right') when data occupies the bottom portion of the chart
• Below chart when dense: fig.legend(loc='lower center', ncol=N, bbox_to_anchor=(0.5, -0.02)) + fig.subplots_adjust(bottom=0.20)
• Never place legend over data area

Date axes — always explicit

Never rely on AutoDateLocator — it overcrowds on multi-year spans:

ax.xaxis.set_major_locator(mdates.MonthLocator(bymonth=[1, 7]))
ax.xaxis.set_major_formatter(mdates.DateFormatter("%b '%y"))
ax.tick_params(axis='x', labelsize=8)

Never auto-open artifacts

Justfile render recipes must not include && open <file>. The user opens files manually.

LaTeX Max-Runs Warning

"WARN: maximum number of runs (9) reached" is cosmetically harmless when there are no \ref{} cross-references in prose. It is caused by fancyhdr + many figures creating layout oscillation. Mitigations:

• Use \needspace not \clearpage (see needspace sizing table in the \needspace section)
• Remove labelformat=empty from \captionsetup — caption numbering churn drives oscillation
• Use htbp float placement rather than forced H

\needspace sizing reference — X = figure height + 1.2in overhead:

BigQuery + Pandas Gotchas (in Quarto Documents)

• BigQuery returns nullable Int64 for integer columns — always .astype('float64') before fillna()
• Quarterly data on a monthly x-axis: df.set_index('quarter').reindex(monthly_idx, method='ffill') + ax.step(..., where='post')
• Never put \n inside BigQuery SQL string literals in Python f-strings — use spaces instead
• Always define intermediate variables BEFORE the Markdown(f"""...""") call — f-strings evaluate at call time, not definition time

Best Practices (TL;DR)

1. Markdown-First: Default to format: gfm with wrap: none for composability, portability, and archival
2. No Line Wrapping: Always use wrap: none for GFM output (avoids artificial line breaks)
3. Dark Mode: Always use auto-dark filter with dual themes for HTML output (accessibility and modern UX)
4. Visual Expression: Use charts and formatted tables, NEVER raw data dumps (df.head(), print(dict))
5. LaTeX for Math: Use LaTeX notation for ALL mathematical expressions ($\alpha = 0.15$, not "alpha = 0.15")
6. Professional Tables: Use LaTeX tables (booktabs) for PDF, Great Tables for HTML
7. No TOC: Table of contents is usually noise - use clear section headings instead
8. PDF for Sharing: Use --to pdf for Google Drive sharing (read-only, professional)
9. Blank Lines Before Lists: ALWAYS include a blank line before every list (bullet or numbered) - no exceptions
10. No Appendix for Sources: Data sources belong in code blocks, not appendix - only add external sources not directly referenced in code to appendix
11. Use Markdown() Class: ALWAYS use Markdown() for text output in code blocks - NEVER use print() or printf() (output must render as formatted markdown)
12. Minimal PDF Titling: For PDF output, suppress YAML title/author/date fields (they produce an academic title block via \maketitle). Use a raw LaTeX minipage inline header instead. Use ## markdown headings for section headings (NOT raw LaTeX \noindent{\large\textbf{...}} blocks — those fight with Quarto's float placement). Exception: only use raw LaTeX headings for the document title line itself.
13. PDF Figure Sizing: Every chart chunk MUST have #| fig-pos: "H", #| fig-width: N, #| fig-height: N, #| out-width: 100%. Always end chunks with plt.close('all'). Set ax.text(...).set_clip_on(True) on all annotation labels. Never use mixed coordinate transforms. See "PDF Figure Sizing — Critical Patterns" section for full details.

PDF Figure Sizing — Critical Patterns

CRITICAL: Matplotlib figure sizing in Quarto PDF output is failure-prone. Follow these rules exactly.

The Working Pattern (copy verbatim)

Every chart chunk that renders to PDF must have ALL of these chunk options:

#| label: fig-my-chart
#| fig-pos: "H"          # force-here via float.sty — prevents deferral/stacking
#| fig-width: 6.5        # must match figsize width in Python code
#| fig-height: 3.5       # must match figsize height in Python code
#| out-width: 100%       # tells LaTeX to scale to full text column width
#| fig-cap: "Caption text with no bare % characters — write 'percent' instead."

And in the Python code:

fig, ax = plt.subplots(figsize=(6.5, 3.5))  # must match chunk fig-width/fig-height
# ... chart code ...
plt.tight_layout()
plt.show()
plt.close('all')   # REQUIRED — prevents state leaking between chunks

rcParams That Must Not Be Changed

plt.rcParams.update({
    'savefig.bbox': None,        # fills declared figsize exactly — do NOT set to 'tight'
    'savefig.pad_inches': 0,
    'figure.dpi': 200,
    'savefig.dpi': 300,
})

savefig.bbox: None is counterintuitive but correct for PDF output. Setting it to 'tight' causes matplotlib to auto-expand the canvas, which fights against the declared figsize and produces malformed figure PDFs.

Root Causes of "Comically Small" Figures

These are the diagnosed failure modes, in order of frequency:

1. Text labels extending beyond xlim/ylim — MOST COMMON

# ❌ BROKEN: annotation text positioned beyond axis limits
for bar, row in zip(bars, df.itertuples()):
    ax.text(bar.get_width() + 4, ..., f"{row.value}")  # +4 may push past xlim
ax.set_xlim(0, 380)  # text at bar.width+4 can exceed 380 → bbox explosion

# ✅ FIX: clip text labels to axes, OR increase xlim to accommodate labels
for bar, row in zip(bars, df.itertuples()):
    t = ax.text(bar.get_width() + 4, ..., f"{row.value}")
    t.set_clip_on(True)   # ← prevents bbox from expanding to include clipped text
# OR: ax.set_xlim(0, 420)  # ensure xlim accommodates largest label

When ax.text() labels are placed at bar.get_width() + offset in data coordinates and those labels extend beyond xlim, the PDF backend measures the full artist bounding box (including out-of-bounds text) when computing the figure's page size. This causes the figure PDF to be output at half or less of the declared figsize — which LaTeX then renders at postage-stamp size even though out-width: 100% is set.

Rule: After setting xlim, add t.set_clip_on(True) to ALL ax.text() calls, or add enough xlim headroom to fit the longest annotation.

2. Mixed coordinate transforms on annotations

# ❌ BROKEN: mixes data coordinates with axis-fraction transform
ax.annotate("", xy=(x0, y0 + 3), xytext=(x1, y1 + 3), ...)  # data coords
ax.text(0.5, max_val + 9, "label", transform=ax.get_xaxis_transform())  # mixed!

# ✅ FIX: use pure axes fraction for floating annotations
ax.annotate("label text",
            xy=(0.5, 0.85), xycoords='axes fraction',
            ha='center', va='center', fontsize=9, ...)

ax.get_xaxis_transform() mixes x=axis-fraction with y=data coordinates. When the y-value in data coords exceeds ylim, the PDF backend's bounding box measurement goes pathological, producing figures that are 10-30× taller than declared. Always use pure coordinate systems — either all data coords or all xycoords='axes fraction'.

3. Missing plt.close('all') between chunks

Without plt.close('all') after each plt.show(), matplotlib figure state (transforms, layout engines, bounding boxes) leaks between Jupyter/Quarto execution chunks. This can cause later charts to inherit corrupt layout state from earlier ones. Always end every chart chunk with:

plt.tight_layout()
plt.show()
plt.close('all')

4. fig-pos: "!ht" instead of "H"

"!ht" (try-here, then top-of-page) causes LaTeX to defer figures when there's insufficient space, stacking them at awkward positions. "H" (force-here via float.sty) places the figure exactly where declared. Requires \usepackage{float} in the LaTeX header.

5. Missing #| fig-width / #| fig-height on chunk

Without explicit chunk-level sizing, Quarto uses YAML defaults and may not pre-allocate the correct float box size before Python renders into it. Always specify both per-chunk.

Diagnosing Figure Size Issues

To identify which figures are malformed without waiting for a full visual review:

# Add keep-tex to render temporarily
cd /path/to/doc && uv run quarto render doc.qmd --to pdf -M keep-tex:true

# Check actual page dimensions of each generated figure PDF
for f in doc_files/figure-pdf/*.pdf; do
  echo -n "$f: "
  pdfinfo "$f" 2>/dev/null | grep "Page size"
done

# A correct 6.5×3.5in figure should be ~468×252 pts (at 72 pts/in)
# A figure with width << 440 pts or height >> 400 pts is malformed

The Nuclear Option

If a figure keeps rendering incorrectly despite all fixes, force PNG raster output:

#| label: fig-problematic
#| fig-pos: "H"
#| dev: png            # ← bypass the PDF vector pipeline entirely
#| dpi: 150
#| fig-width: 6.5
#| fig-height: 3.5
#| out-width: 100%

PNG output is immune to all the bbox/transform issues because matplotlib renders to a fixed-size raster and Quarto embeds it directly. Use as a last resort since vector PDF is crisper.

Caption Numbering

% Restore default numbering (Figure 1., Figure 2., etc.) with bold prefix:
\captionsetup{font={small,it},justification=centering,skip=6pt,labelfont=bf}

% Suppress numbering (caption text only, no "Figure N." prefix):
\captionsetup{font={small,it},justification=centering,skip=6pt,labelformat=empty,labelsep=none}

Never use bare % in fig-cap strings — write "percent" or "percentage points" instead. LaTeX may fail to compile depending on pandoc version.

Reference Lines (axvline/axhline) Opacity

Reference lines (baselines, averages) should be visible context, not dominant elements. Always set alpha=0.4:

ax.axvline(48.9, color=NAVY,  linestyle=":",  linewidth=1.6, alpha=0.4, label="Baseline", zorder=3)
ax.axhline(37.8, color=SLATE, linestyle="--", linewidth=1.4, alpha=0.4, label="Avg", zorder=3)

At alpha=1.0 (default), reference lines dominate the chart and compete with the data bars. alpha=0.4 keeps them readable without visual dominance.

Visual Expression Philosophy

CRITICAL: Quarto documents are for COMMUNICATION, not raw data dumps.

Quarto outputs are static documents meant to convey insights to humans. Raw dataframes, print statements, and JSON blobs fail to communicate effectively.

Visual Hierarchy (Use in Order)

1. Charts/Plots - For trends, distributions, comparisons, relationships
2. Formatted Tables - For structured data with styling and context
3. Formatted Metrics - For key numbers with context and formatting
4. Raw Output - NEVER (not even for debugging - use separate analysis files)

Anti-Patterns: What NOT to Do

# ❌ BAD: Raw dataframe dump
df.head()

# ❌ BAD: Print statements - output renders as plain text, not formatted markdown
print(f"Total: {total}")
print(data_dict)

# ❌ BAD: printf/print for text - use Markdown() instead
print("## Summary\n- Item 1\n- Item 2")  # Renders as plain text!

# ❌ BAD: Bare variable returning data structure
result  # Returns raw dict/JSON

# ❌ BAD: DataFrame info without formatting
df.describe()
df.info()

# ✅ GOOD: Use Markdown() for ALL text output
from IPython.display import Markdown
Markdown(f"""
## Summary

- **Total**: {total:,}
- **Average**: {avg:.2f}
""")

❌ BAD: Plain text for mathematical notation
- The growth rate is alpha = 0.15 or 15%
- We calculated the mean mu = sum(xi)/n
- The correlation coefficient r = 0.85

❌ BAD: No table formatting
```python
print(df.head())

❌ BAD: Using asterisks for equations

• E = m * c^2
• y = beta0 + beta1 * x

### Good Patterns: Visual Communication

```python
# ✅ GOOD: Chart for trends
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(10, 6))
df.groupby('date')['sales'].sum().plot(ax=ax, kind='line')
ax.set_title('Sales Trend Over Time')
ax.set_ylabel('Sales ($)')
plt.tight_layout()
plt.show()

# ✅ GOOD: Formatted table using Great Tables
from great_tables import GT

(GT(df.head(10))
    .tab_header(title="Top 10 Sales Records")
    .fmt_currency(columns="sales", currency="USD")
    .fmt_date(columns="date", date_style="medium"))

# ✅ GOOD: Formatted table using pandas markdown
from IPython.display import Markdown
Markdown(df.head(10).to_markdown(index=False, tablefmt='pipe'))

# ✅ GOOD: Formatted metrics in markdown with LaTeX
from IPython.display import Markdown

Markdown(f"""
## Key Metrics

- **Total Sales**: ${total_sales:,.2f}
- **Average Order**: ${avg_order:,.2f}
- **Growth Rate**: $\\alpha = {growth_rate:.1%}$ (15% YoY)
- **Top Product**: {top_product}

### Statistical Summary

The linear regression model $y = \\beta_0 + \\beta_1 x + \\epsilon$ yielded:

- Slope: $\\hat{{\\beta_1}} = 3.2$ (SE = 0.4)
- $R^2 = 0.78$, indicating strong fit
""")

# NOTE: Mermaid diagrams must use native Quarto syntax outside Python blocks
# Use ```{mermaid} directly in markdown, NOT inside Markdown() calls

Why Visual Expression Matters

• Documents are for humans: Show insights, not data structures
• Static format: No interactive exploration - must communicate clearly on first view
• Traceable reasoning: Visualize fact→conclusion chains, not raw JSON
• Professional output: Charts and tables look polished in PDF/HTML/Word
• Accessibility: Visual hierarchy helps readers navigate content
• Shareability: Well-formatted outputs communicate without explanation

Narrative Structure

Build understanding progressively through a series of sections:

Document Flow

1. Abstract - The punchline first (executive summary for busy readers)
2. Key Findings - Scannable bullet points with confidence levels
3. Base Facts - Individual observations/data, each in its own section
4. Synthesis Sections - Combine earlier facts into higher-level insights
5. Conclusion - Final synthesis referencing the insights above

Base Facts (Individual Sections)

Each base fact is an independent observation with its own data and evidence. Use descriptive headers (not "Fact 1"):

## Response Time Distribution

\needspace{3in}

Analysis of the past 7 days shows significant tail latency:
- p50: 45ms
- p95: 230ms  
- p99: 890ms (concerning)

```{python}
#| echo: false
# Chart showing latency distribution

### Synthesis Sections

Synthesis sections **explicitly reference** which earlier sections they build upon:

```qmd
## Performance Degradation Under Load

\needspace{4in}

Building on the response time distribution and traffic patterns above, we 
observe a clear correlation: p99 latency spikes to 2.3s during the 2-4pm 
peak traffic window. The system handles baseline load well but degrades 
significantly under peak conditions.

Keeping Content Together (PDF)

Use \needspace{Xin} before sections with charts/diagrams to prevent awkward page breaks and large whitespace gaps:

## Revenue by Carrier

\needspace{4in}

```{python}
# Chart code here

**Guidelines:**
- **Mermaid diagrams**: `\needspace{2in}`
- **Single chart**: `\needspace{3in}`
- **Chart + explanation**: `\needspace{4in}`

**Requires** in YAML frontmatter:
```yaml
format:
  pdf:
    include-in-header:
      text: |
        \usepackage{needspace}

Markdown Formatting Rules (CRITICAL)

IMPORTANT: Markdown requires blank lines between paragraphs for proper rendering.

Line Breaks and Paragraphs

The Problem:

❌ BAD: This will render as one long line
This text appears on a new line in the source
But it renders on the same line as above
Because there's no blank line between them

The Solution:

✅ GOOD: This renders as separate paragraphs

This text appears on its own line because there's a blank line above it.

Each paragraph needs a blank line before and after it.

Common Markdown Patterns

Paragraphs (need blank lines):

This is paragraph one.

This is paragraph two.

This is paragraph three.

Lists REQUIRE blank line before the list:

❌ BAD: List doesn't render correctly
Here is some text:
- Item 1
- Item 2

✅ GOOD: Blank line before list
Here is some text:

- Item 1
- Item 2

Numbered lists also require blank line before:

❌ BAD: Numbered list broken
The steps are:
1. First step
2. Second step

✅ GOOD: Blank line before numbered list
The steps are:

1. First step
2. Second step

Lists (no blank lines between items):

- Item 1
- Item 2
- Item 3

Multi-paragraph list items (blank lines within item):

- Item 1 with first paragraph

  Item 1 continued with second paragraph (indented 2 spaces)

- Item 2 starts here

Headers (blank line before and after):

Previous paragraph ends here.

## Section Header

New paragraph starts here.

Code blocks (blank line before and after):

Previous paragraph ends here.

```python
print("code block")
```

New paragraph starts here.

Quarto-Specific Markdown

Python code output:

#| echo: false
from IPython.display import Markdown

# ❌ BAD: Single newline won't create paragraph break
Markdown("Line 1\nLine 2")  # Renders as: Line 1 Line 2

# ✅ GOOD: Double newline creates paragraph break
Markdown("Line 1\n\nLine 2")  # Renders as separate paragraphs

# ✅ GOOD: Use triple-quoted string with blank lines
Markdown("""
Paragraph one.

Paragraph two.

Paragraph three.
""")

F-strings in Markdown:

#| echo: false
from IPython.display import Markdown

# ✅ GOOD: Blank lines between paragraphs
Markdown(f"""
## Analysis Results

The growth rate is {growth_rate:.1%}.

This represents a significant increase over last quarter.

We recommend increasing inventory by {inventory_increase:,} units.
""")

Best Practices

✅ DO:

• Use blank lines between all paragraphs
• Use blank lines before and after headers
• Use blank lines before and after code blocks
• Use blank lines before and after tables
• Use triple-quoted strings for multi-line markdown in Python

❌ DON'T:

• Use single newlines and expect paragraph breaks
• Forget blank lines around headers or code blocks
• Mix single and double newlines inconsistently
• Use \n in strings expecting paragraph breaks (use \n\n)

Testing Markdown Formatting

Quick test:

# Create test document
cat > test.qmd << 'EOF'
---
title: "Markdown Test"
format: gfm
---

Paragraph 1.

Paragraph 2.

## Header

Paragraph 3.
EOF

# Render and check output
quarto render test.qmd --to gfm
cat test.md

LLM Self-Reasoning with Quarto

Use /think command for structured analysis with graduated detail.

Document Structure

Documents use graduated detail so readers can stop at their desired depth:

Abstract (paragraph)

• Self-contained executive summary
• Complete story: what, why, result, meaning
• Include key metrics and confidence assessment

Key Findings (3-5 bullets)

• Result + confidence + brief evidence
• Scannable - each finding valuable standalone
• Format: **[Finding]**: [Result] — [Evidence] (Confidence)

Investigation (detailed)

• Observations (sourced facts with academic citations [source])
• Analysis (visual reasoning, statistical evidence)
• Interpretation (what it means, confidence, dependencies)

Appendix (optional)

• Investigation notes, dead ends, debugging traces
• External data sources NOT directly referenced in code blocks (e.g., verbal conversations, meeting notes, prior analyses)
• Do NOT duplicate data sources already expressed in code blocks - the code IS the source documentation

Visual Evidence

Include diagram, chart, or table for most findings:

• Mermaid diagrams for reasoning flows
• Charts for quantitative analysis
• Formatted tables for comparative data

Example Invocation

/think Why is the login endpoint returning 500 errors intermittently?

Creates analysis document with abstract-first structure, key findings with confidence levels, and visual reasoning chains.

See /think command for full template.

When to Use Quarto

Perfect for:

• LLM epistemological reasoning (use /think command)
• Static reports and documentation (no interactivity needed)
• Multi-format publishing (PDF + HTML + Word from single source)
• Scientific documents (equations, citations, cross-references)
• Presentations (RevealJS HTML slides, PowerPoint, Beamer PDF)
• Websites and blogs (multi-page projects)
• Exporting Jupyter notebooks for publication

NOT for:

• Interactive dashboards (use Shiny or dedicated dashboard tools)
• Real-time data updates (use web dashboards)
• When you need widgets/sliders for end users

Data Provenance and Portability

CRITICAL: Quarto documents must be reproducible with documented dependencies.

When someone runs quarto render analysis.qmd, they should be able to get identical results given:

• The document itself (.qmd file)
• Documented external dependencies (with setup instructions)
• Access to the same data sources (APIs, databases)

The Portability Contract

A .qmd file defines a complete data pipeline. The document contains:

1. Data extraction logic - How to obtain the data (queries, API calls, etc.)
2. Transformation code - How to process and analyze
3. Presentation - Charts, tables, narrative
4. Dependency documentation - Setup instructions for heavy dependencies

Practical limits: Some dependencies are too expensive to rebuild on every render:

• Vector indexes (LanceDB, FAISS) - Document how to build, reference existing
• Large datasets - Commit to git or document extraction, don't re-download
• ML models - Reference by path with setup instructions

The key is documentation: readers must understand what's needed and how to set it up.

Anti-Pattern: Opaque File References

❌ BAD: Referencing local files without provenance

df = pd.read_json('/tmp/orders.jsonl', lines=True)  # Where did this come from?
df = pd.read_csv('sales.csv')  # Who created this? When? How?

# "This JSONL file" without explaining its origin
data = load_data('extracted_metrics.jsonl')  # Non-portable!

Good Pattern: Embed Data Extraction

✅ PREFERRED: Document defines where data comes from

#| cache: true
import pandas as pd
from epq import bq

# Extract from BigQuery via client library — cached to avoid re-running on every render
chunks = list(bq.run_bq_query("""
    SELECT * FROM production.orders
    WHERE date >= '2024-01-01'
    AND status = 'completed'
"""))
df = pd.concat(chunks, ignore_index=True)

✅ ALSO GOOD: Canonical external sources

#| cache: true
import pandas as pd

# Public dataset with stable URL
df = pd.read_csv('https://data.company.com/public/sales-2024.csv')

# Or versioned data in the same repository
df = pd.read_csv('data/sales-2024-v2.csv')  # Committed to git with the .qmd

Heavy Dependencies: Document, Don't Rebuild

Rule of thumb: Un-cached renders should complete in < 60 seconds.

• < 60 seconds → Embed in document (with cache: true)
• > 60 seconds → Document as external dependency with setup instructions

Some dependencies are too expensive to recreate on every render. Document them clearly so readers can set up the environment.

✅ GOOD: Reference with setup documentation

#| echo: false
import subprocess

# DEPENDENCY: LanceDB index at ~/.lancedb/documents
# Setup: lancer ingest -t documents ~/corpus/*.md
# This index contains ~50k documents and takes ~10 min to build

result = subprocess.run(
    ['lancer', 'search', '-t', 'documents', 'shipping rate errors', '--limit', '20'],
    capture_output=True, text=True, check=True
)
relevant_docs = result.stdout

✅ GOOD: Prerequisites section in document

---
title: "Knowledge Base Analysis"
---

## Prerequisites

This analysis requires the following setup:

1. **LanceDB index**: `lancer ingest -t documents ~/corpus/*.md`
2. **BigQuery access**: Authenticated via `gcloud auth application-default login`
3. **Data snapshot**: Run `./scripts/extract-data.sh` (takes ~5 min)

## Analysis
...

❌ BAD: Silent dependency on local state

# No documentation about what this index is or how to create it
results = lancer.search("documents", "query")  # Will fail for anyone else

Caching for Iteration Speed

Use cache: true to avoid re-running expensive operations during iteration.

Requires jupyter-cache (one-time install):

uv add jupyter-cache

Per-cell caching:

#| cache: true
#| label: data-extraction

# This cell only re-executes if the code changes
from epq import bq
import pandas as pd
chunks = list(bq.run_bq_query("SELECT ..."))
df = pd.concat(chunks, ignore_index=True)

Document-wide caching in YAML frontmatter:

---
title: "Analysis Report"
execute:
  cache: true
---

Freeze for Project-Level Caching

For projects with many documents, use freeze to cache execution results in version control:

# _quarto.yml (project config)
execute:
  freeze: auto  # Re-render only when source changes

Key difference:

• cache: true - Caches cell outputs locally (Jupyter Cache)
• freeze: auto - Stores results in _freeze/ directory (can commit to git)

When to use freeze:

• Large projects with many collaborators
• Documents with environment-specific dependencies
• When you want cached results portable across machines (commit _freeze/)

Cache-Read-or-Query Pattern (BigQuery / Expensive APIs)

Use this pattern when cache: true is insufficient — specifically when:

• Querying BigQuery or other expensive external APIs
• Cache must survive Quarto kernel restarts (Jupyter cache does not)
• You want explicit control over cache invalidation (not tied to source changes)
• Cache files are machine-specific and should be gitignored

Set execute: cache: false in frontmatter when using this pattern (disable Jupyter cache to avoid double-caching).

Helper functions (add once per document, in a setup cell):

import json
from pathlib import Path
from datetime import datetime, timezone

_CACHE_DIR = Path('data/cache')
_CACHE_DIR.mkdir(parents=True, exist_ok=True)

def _cache_path(name):
    return _CACHE_DIR / f"{name}.json"

def _read_cache(name):
    p = _cache_path(name)
    if p.exists():
        return json.loads(p.read_text())
    return None

def _write_cache(name, records, scalars=None):
    p = _cache_path(name)
    p.write_text(json.dumps({
        '_queried_at': datetime.now(timezone.utc).isoformat(),
        'records': records,
        'scalars': scalars or {}
    }, default=str))

Per-dataset usage template (repeat for each dataset):

_c = _read_cache('my_dataset')
if _c:
    df = pd.DataFrame(_c['records'])
    my_scalar = float(_c['scalars']['my_scalar'])
else:
    df = run_bq_query(my_query)
    my_scalar = float(df['col'].values[0])
    _write_cache('my_dataset', df.to_dict(orient='records'), {
        'my_scalar': my_scalar,
    })

Cache hit → reads DataFrame and scalars from JSON; no BigQuery call. Cache miss → queries BigQuery live, writes cache, continues render. Query failure on miss → render fails loudly (intentional — no silent fallback).

Never do:

• except Exception: my_scalar = 42 — silent fallback masks broken queries
• Assign a constant inside try: without a preceding query call — hidden constant, not a live value
• Omit _write_cache() in the else branch — next render re-queries unnecessarily

Cache file format:

{
    "_queried_at": "2026-02-20T16:00:00Z",
    "records": [...],
    "scalars": {...}
}

Serialization notes:

• Numpy arrays → store as lists (df['col'].tolist()), reconstruct with np.array(...)
• Dates → use default=str in json.dumps to handle non-serializable types

Cache invalidation:

# .gitignore
data/cache/

# Justfile recipe
delete-cache:
    rm -rf data/cache/

Complete Example: Reproducible Analysis

---
title: "Q4 2024 Sales Analysis"
author: "Josh Lane"
date: "2024-12-31"
format:
  gfm:
    wrap: none
  html:
    theme:
      dark: darkly
      light: flatly
execute:
  cache: true
filters:
  - auto-dark
---

## Data Extraction

```{python}
#| cache: true
#| label: extract-sales
import subprocess
import io
import pandas as pd

# Reproducible: Query is embedded in the document
result = subprocess.run([
    'bigquery', 'query',
    '''
    SELECT date, product, region, sales, units
    FROM production.sales
    WHERE EXTRACT(QUARTER FROM date) = 4
      AND EXTRACT(YEAR FROM date) =

…(truncated)