` without breaking Pandoc ## PDF Figure Rules ### Split multi-story panels If a combined figure has two sub-panels telling different insights, split into separate `fig-*` cells with their own captions. Ask upfront whether panels should be split — this is cheaper than rework after the fact. ### Legend placement - Top-right (`loc='upper right'`) when data occupies the bottom portion of the chart - Below chart when dense: `fig.legend(loc='lower center', ncol=N, bbox_to_anchor=(0.5, -0.02))` + `fig.subplots_adjust(bottom=0.20)` - Never place legend over data area ### Date axes — always explicit Never rely on `AutoDateLocator` — it overcrowds on multi-year spans: ```python ax.xaxis.set_major_locator(mdates.MonthLocator(bymonth=[1, 7])) ax.xaxis.set_major_formatter(mdates.DateFormatter("%b '%y")) ax.tick_params(axis='x', labelsize=8) ``` ### Never auto-open artifacts Justfile render recipes must not include `&& open <file>`. The user opens files manually. ## LaTeX Max-Runs Warning "WARN: maximum number of runs (9) reached" is **cosmetically harmless** when there are no `\ref{}` cross-references in prose. It is caused by `fancyhdr` + many figures creating layout oscillation. Mitigations: - Use `\needspace` not `\clearpage` (see needspace sizing table in the `\needspace` section) - Remove `labelformat=empty` from `\captionsetup` — caption numbering churn drives oscillation - Use `htbp` float placement rather than forced `H` `\needspace` sizing reference — X = figure height + 1.2in overhead: | Figure height | `\needspace` | |---|---| | 3.2in | `\needspace{4.5in}` | | 4.0in | `\needspace{5.2in}` | | 5.5in | `\needspace{6.8in}` | | Prose only | `\needspace{2.5in}` | ## BigQuery + Pandas Gotchas (in Quarto Documents) - BigQuery returns nullable `Int64` for integer columns — always `.astype('float64')` before `fillna()` - Quarterly data on a monthly x-axis: `df.set_index('quarter').reindex(monthly_idx, method='ffill')` + `ax.step(..., where='post')` - Never put `\n` inside BigQuery SQL string literals in Python f-strings — use spaces instead - Always define intermediate variables BEFORE the `Markdown(f"""...""")` call — f-strings evaluate at call time, not definition time --- ## Best Practices (TL;DR) 1. **Markdown-First**: Default to `format: gfm` with `wrap: none` for composability, portability, and archival 2. **No Line Wrapping**: Always use `wrap: none` for GFM output (avoids artificial line breaks) 3. **Dark Mode**: Always use `auto-dark` filter with dual themes for HTML output (accessibility and modern UX) 4. **Visual Expression**: Use charts and formatted tables, NEVER raw data dumps (`df.head()`, `print(dict)`) 5. **LaTeX for Math**: Use LaTeX notation for ALL mathematical expressions ($\alpha = 0.15$, not "alpha = 0.15") 6. **Professional Tables**: Use LaTeX tables (booktabs) for PDF, Great Tables for HTML 7. **No TOC**: Table of contents is usually noise - use clear section headings instead 8. **PDF for Sharing**: Use `--to pdf` for Google Drive sharing (read-only, professional) 9. **Blank Lines Before Lists**: ALWAYS include a blank line before every list (bullet or numbered) - no exceptions 10. **No Appendix for Sources**: Data sources belong in code blocks, not appendix - only add external sources not directly referenced in code to appendix 11. **Use Markdown() Class**: ALWAYS use `Markdown()` for text output in code blocks - NEVER use `print()` or `printf()` (output must render as formatted markdown) 12. **Minimal PDF Titling**: For PDF output, suppress YAML `title`/`author`/`date` fields (they produce an academic title block via `\maketitle`). Use a raw LaTeX minipage inline header instead. Use `##` markdown headings for section headings (NOT raw LaTeX `\noindent{\large\textbf{...}}` blocks — those fight with Quarto's float placement). Exception: only use raw LaTeX headings for the document title line itself. 13. **PDF Figure Sizing**: Every chart chunk MUST have `#| fig-pos: "H"`, `#| fig-width: N`, `#| fig-height: N`, `#| out-width: 100%`. Always end chunks with `plt.close('all')`. Set `ax.text(...).set_clip_on(True)` on all annotation labels. Never use mixed coordinate transforms. See "PDF Figure Sizing — Critical Patterns" section for full details. ## PDF Figure Sizing — Critical Patterns **CRITICAL: Matplotlib figure sizing in Quarto PDF output is failure-prone. Follow these rules exactly.** ### The Working Pattern (copy verbatim) Every chart chunk that renders to PDF must have ALL of these chunk options: ```python #| label: fig-my-chart #| fig-pos: "H" # force-here via float.sty — prevents deferral/stacking #| fig-width: 6.5 # must match figsize width in Python code #| fig-height: 3.5 # must match figsize height in Python code #| out-width: 100% # tells LaTeX to scale to full text column width #| fig-cap: "Caption text with no bare % characters — write 'percent' instead." ``` And in the Python code: ```python fig, ax = plt.subplots(figsize=(6.5, 3.5)) # must match chunk fig-width/fig-height # ... chart code ... plt.tight_layout() plt.show() plt.close('all') # REQUIRED — prevents state leaking between chunks ``` ### rcParams That Must Not Be Changed ```python plt.rcParams.update({ 'savefig.bbox': None, # fills declared figsize exactly — do NOT set to 'tight' 'savefig.pad_inches': 0, 'figure.dpi': 200, 'savefig.dpi': 300, }) ``` **`savefig.bbox: None`** is counterintuitive but correct for PDF output. Setting it to `'tight'` causes matplotlib to auto-expand the canvas, which fights against the declared figsize and produces malformed figure PDFs. ### Root Causes of "Comically Small" Figures These are the diagnosed failure modes, in order of frequency: **1. Text labels extending beyond xlim/ylim — MOST COMMON** ```python # ❌ BROKEN: annotation text positioned beyond axis limits for bar, row in zip(bars, df.itertuples()): ax.text(bar.get_width() + 4, ..., f"{row.value}") # +4 may push past xlim ax.set_xlim(0, 380) # text at bar.width+4 can exceed 380 → bbox explosion # ✅ FIX: clip text labels to axes, OR increase xlim to accommodate labels for bar, row in zip(bars, df.itertuples()): t = ax.text(bar.get_width() + 4, ..., f"{row.value}") t.set_clip_on(True) # ← prevents bbox from expanding to include clipped text # OR: ax.set_xlim(0, 420) # ensure xlim accommodates largest label ``` When `ax.text()` labels are placed at `bar.get_width() + offset` in data coordinates and those labels extend beyond `xlim`, the PDF backend measures the full artist bounding box (including out-of-bounds text) when computing the figure's page size. This causes the figure PDF to be output at **half or less of the declared figsize** — which LaTeX then renders at postage-stamp size even though `out-width: 100%` is set. **Rule:** After setting `xlim`, add `t.set_clip_on(True)` to ALL `ax.text()` calls, or add enough xlim headroom to fit the longest annotation. **2. Mixed coordinate transforms on annotations** ```python # ❌ BROKEN: mixes data coordinates with axis-fraction transform ax.annotate("", xy=(x0, y0 + 3), xytext=(x1, y1 + 3), ...) # data coords ax.text(0.5, max_val + 9, "label", transform=ax.get_xaxis_transform()) # mixed! # ✅ FIX: use pure axes fraction for floating annotations ax.annotate("label text", xy=(0.5, 0.85), xycoords='axes fraction', ha='center', va='center', fontsize=9, ...) ``` `ax.get_xaxis_transform()` mixes x=axis-fraction with y=data coordinates. When the y-value in data coords exceeds ylim, the PDF backend's bounding box measurement goes pathological, producing figures that are 10-30× taller than declared. **Always use pure coordinate systems** — either all data coords or all `xycoords='axes fraction'`. **3. Missing `plt.close('all')` between chunks** Without `plt.close('all')` after each `plt.show()`, matplotlib figure state (transforms, layout engines, bounding boxes) leaks between Jupyter/Quarto execution chunks. This can cause later charts to inherit corrupt layout state from earlier ones. Always end every chart chunk with: ```python plt.tight_layout() plt.show() plt.close('all') ``` **4. `fig-pos: "!ht"` instead of `"H"`** `"!ht"` (try-here, then top-of-page) causes LaTeX to defer figures when there's insufficient space, stacking them at awkward positions. `"H"` (force-here via `float.sty`) places the figure exactly where declared. Requires `\usepackage{float}` in the LaTeX header. **5. Missing `#| fig-width` / `#| fig-height` on chunk** Without explicit chunk-level sizing, Quarto uses YAML defaults and may not pre-allocate the correct float box size before Python renders into it. Always specify both per-chunk. ### Diagnosing Figure Size Issues To identify which figures are malformed without waiting for a full visual review: ```bash # Add keep-tex to render temporarily cd /path/to/doc && uv run quarto render doc.qmd --to pdf -M keep-tex:true # Check actual page dimensions of each generated figure PDF for f in doc_files/figure-pdf/*.pdf; do echo -n "$f: " pdfinfo "$f" 2>/dev/null | grep "Page size" done # A correct 6.5×3.5in figure should be ~468×252 pts (at 72 pts/in) # A figure with width << 440 pts or height >> 400 pts is malformed ``` ### The Nuclear Option If a figure keeps rendering incorrectly despite all fixes, force PNG raster output: ```python #| label: fig-problematic #| fig-pos: "H" #| dev: png # ← bypass the PDF vector pipeline entirely #| dpi: 150 #| fig-width: 6.5 #| fig-height: 3.5 #| out-width: 100% ``` PNG output is immune to all the bbox/transform issues because matplotlib renders to a fixed-size raster and Quarto embeds it directly. Use as a last resort since vector PDF is crisper. ### Caption Numbering ```latex % Restore default numbering (Figure 1., Figure 2., etc.) with bold prefix: \captionsetup{font={small,it},justification=centering,skip=6pt,labelfont=bf} % Suppress numbering (caption text only, no "Figure N." prefix): \captionsetup{font={small,it},justification=centering,skip=6pt,labelformat=empty,labelsep=none} ``` **Never use bare `%` in `fig-cap` strings** — write "percent" or "percentage points" instead. LaTeX may fail to compile depending on pandoc version. ### Reference Lines (axvline/axhline) Opacity Reference lines (baselines, averages) should be visible context, not dominant elements. Always set `alpha=0.4`: ```python ax.axvline(48.9, color=NAVY, linestyle=":", linewidth=1.6, alpha=0.4, label="Baseline", zorder=3) ax.axhline(37.8, color=SLATE, linestyle="--", linewidth=1.4, alpha=0.4, label="Avg", zorder=3) ``` At `alpha=1.0` (default), reference lines dominate the chart and compete with the data bars. `alpha=0.4` keeps them readable without visual dominance. --- ## Visual Expression Philosophy **CRITICAL: Quarto documents are for COMMUNICATION, not raw data dumps.** Quarto outputs are static documents meant to convey insights to humans. Raw dataframes, print statements, and JSON blobs fail to communicate effectively. ### Visual Hierarchy (Use in Order) 1. **Charts/Plots** - For trends, distributions, comparisons, relationships 2. **Formatted Tables** - For structured data with styling and context 3. **Formatted Metrics** - For key numbers with context and formatting 4. **Raw Output** - NEVER (not even for debugging - use separate analysis files) ### Anti-Patterns: What NOT to Do ```python # ❌ BAD: Raw dataframe dump df.head() # ❌ BAD: Print statements - output renders as plain text, not formatted markdown print(f"Total: {total}") print(data_dict) # ❌ BAD: printf/print for text - use Markdown() instead print("## Summary\n- Item 1\n- Item 2") # Renders as plain text! # ❌ BAD: Bare variable returning data structure result # Returns raw dict/JSON # ❌ BAD: DataFrame info without formatting df.describe() df.info() # ✅ GOOD: Use Markdown() for ALL text output from IPython.display import Markdown Markdown(f""" ## Summary - **Total**: {total:,} - **Average**: {avg:.2f} """) ``` ```markdown ❌ BAD: Plain text for mathematical notation - The growth rate is alpha = 0.15 or 15% - We calculated the mean mu = sum(xi)/n - The correlation coefficient r = 0.85 ❌ BAD: No table formatting ```python print(df.head()) ``` ❌ BAD: Using asterisks for equations - E = m * c^2 - y = beta0 + beta1 * x ``` ### Good Patterns: Visual Communication ```python # ✅ GOOD: Chart for trends import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(10, 6)) df.groupby('date')['sales'].sum().plot(ax=ax, kind='line') ax.set_title('Sales Trend Over Time') ax.set_ylabel('Sales ($)') plt.tight_layout() plt.show() # ✅ GOOD: Formatted table using Great Tables from great_tables import GT (GT(df.head(10)) .tab_header(title="Top 10 Sales Records") .fmt_currency(columns="sales", currency="USD") .fmt_date(columns="date", date_style="medium")) # ✅ GOOD: Formatted table using pandas markdown from IPython.display import Markdown Markdown(df.head(10).to_markdown(index=False, tablefmt='pipe')) # ✅ GOOD: Formatted metrics in markdown with LaTeX from IPython.display import Markdown Markdown(f""" ## Key Metrics - **Total Sales**: ${total_sales:,.2f} - **Average Order**: ${avg_order:,.2f} - **Growth Rate**: $\\alpha = {growth_rate:.1%}$ (15% YoY) - **Top Product**: {top_product} ### Statistical Summary The linear regression model $y = \\beta_0 + \\beta_1 x + \\epsilon$ yielded: - Slope: $\\hat{{\\beta_1}} = 3.2$ (SE = 0.4) - $R^2 = 0.78$, indicating strong fit """) # NOTE: Mermaid diagrams must use native Quarto syntax outside Python blocks # Use ```{mermaid} directly in markdown, NOT inside Markdown() calls ``` ### Why Visual Expression Matters - **Documents are for humans**: Show insights, not data structures - **Static format**: No interactive exploration - must communicate clearly on first view - **Traceable reasoning**: Visualize fact→conclusion chains, not raw JSON - **Professional output**: Charts and tables look polished in PDF/HTML/Word - **Accessibility**: Visual hierarchy helps readers navigate content - **Shareability**: Well-formatted outputs communicate without explanation ## Narrative Structure **Build understanding progressively through a series of sections:** ### Document Flow 1. **Abstract** - The punchline first (executive summary for busy readers) 2. **Key Findings** - Scannable bullet points with confidence levels 3. **Base Facts** - Individual observations/data, each in its own section 4. **Synthesis Sections** - Combine earlier facts into higher-level insights 5. **Conclusion** - Final synthesis referencing the insights above ### Base Facts (Individual Sections) Each base fact is an independent observation with its own data and evidence. Use descriptive headers (not "Fact 1"): ```qmd ## Response Time Distribution \needspace{3in} Analysis of the past 7 days shows significant tail latency: - p50: 45ms - p95: 230ms - p99: 890ms (concerning) ```{python} #| echo: false # Chart showing latency distribution ``` ``` ### Synthesis Sections Synthesis sections **explicitly reference** which earlier sections they build upon: ```qmd ## Performance Degradation Under Load \needspace{4in} Building on the response time distribution and traffic patterns above, we observe a clear correlation: p99 latency spikes to 2.3s during the 2-4pm peak traffic window. The system handles baseline load well but degrades significantly under peak conditions. ``` ### Keeping Content Together (PDF) Use `\needspace{Xin}` before sections with charts/diagrams to prevent awkward page breaks and large whitespace gaps: ```qmd ## Revenue by Carrier \needspace{4in} ```{python} # Chart code here ``` ``` **Guidelines:** - **Mermaid diagrams**: `\needspace{2in}` - **Single chart**: `\needspace{3in}` - **Chart + explanation**: `\needspace{4in}` **Requires** in YAML frontmatter: ```yaml format: pdf: include-in-header: text: | \usepackage{needspace} ``` ## Markdown Formatting Rules (CRITICAL) **IMPORTANT: Markdown requires blank lines between paragraphs for proper rendering.** ### Line Breaks and Paragraphs **The Problem:** ```markdown ❌ BAD: This will render as one long line This text appears on a new line in the source But it renders on the same line as above Because there's no blank line between them ``` **The Solution:** ```markdown ✅ GOOD: This renders as separate paragraphs This text appears on its own line because there's a blank line above it. Each paragraph needs a blank line before and after it. ``` ### Common Markdown Patterns **Paragraphs (need blank lines):** ```markdown This is paragraph one. This is paragraph two. This is paragraph three. ``` **Lists REQUIRE blank line before the list:** ```markdown ❌ BAD: List doesn't render correctly Here is some text: - Item 1 - Item 2 ✅ GOOD: Blank line before list Here is some text: - Item 1 - Item 2 ``` **Numbered lists also require blank line before:** ```markdown ❌ BAD: Numbered list broken The steps are: 1. First step 2. Second step ✅ GOOD: Blank line before numbered list The steps are: 1. First step 2. Second step ``` **Lists (no blank lines between items):** ```markdown - Item 1 - Item 2 - Item 3 ``` **Multi-paragraph list items (blank lines within item):** ```markdown - Item 1 with first paragraph Item 1 continued with second paragraph (indented 2 spaces) - Item 2 starts here ``` **Headers (blank line before and after):** ```markdown Previous paragraph ends here. ## Section Header New paragraph starts here. ``` **Code blocks (blank line before and after):** ````markdown Previous paragraph ends here. ```python print("code block") ``` New paragraph starts here. ```` ### Quarto-Specific Markdown **Python code output:** ```python #| echo: false from IPython.display import Markdown # ❌ BAD: Single newline won't create paragraph break Markdown("Line 1\nLine 2") # Renders as: Line 1 Line 2 # ✅ GOOD: Double newline creates paragraph break Markdown("Line 1\n\nLine 2") # Renders as separate paragraphs # ✅ GOOD: Use triple-quoted string with blank lines Markdown(""" Paragraph one. Paragraph two. Paragraph three. """) ``` **F-strings in Markdown:** ```python #| echo: false from IPython.display import Markdown # ✅ GOOD: Blank lines between paragraphs Markdown(f""" ## Analysis Results The growth rate is {growth_rate:.1%}. This represents a significant increase over last quarter. We recommend increasing inventory by {inventory_increase:,} units. """) ``` ### Best Practices ✅ **DO:** - Use blank lines between all paragraphs - Use blank lines before and after headers - Use blank lines before and after code blocks - Use blank lines before and after tables - Use triple-quoted strings for multi-line markdown in Python ❌ **DON'T:** - Use single newlines and expect paragraph breaks - Forget blank lines around headers or code blocks - Mix single and double newlines inconsistently - Use `\n` in strings expecting paragraph breaks (use `\n\n`) ### Testing Markdown Formatting **Quick test:** ```bash # Create test document cat > test.qmd << 'EOF' --- title: "Markdown Test" format: gfm --- Paragraph 1. Paragraph 2. ## Header Paragraph 3. EOF # Render and check output quarto render test.qmd --to gfm cat test.md ``` ## LLM Self-Reasoning with Quarto Use `/think` command for structured analysis with graduated detail. ### Document Structure Documents use graduated detail so readers can stop at their desired depth: **Abstract** (paragraph) - Self-contained executive summary - Complete story: what, why, result, meaning - Include key metrics and confidence assessment **Key Findings** (3-5 bullets) - Result + confidence + brief evidence - Scannable - each finding valuable standalone - Format: `**[Finding]**: [Result] — [Evidence] (Confidence)` **Investigation** (detailed) - Observations (sourced facts with academic citations `[source]`) - Analysis (visual reasoning, statistical evidence) - Interpretation (what it means, confidence, dependencies) **Appendix** (optional) - Investigation notes, dead ends, debugging traces - External data sources NOT directly referenced in code blocks (e.g., verbal conversations, meeting notes, prior analyses) - Do NOT duplicate data sources already expressed in code blocks - the code IS the source documentation ### Visual Evidence Include diagram, chart, or table for most findings: - Mermaid diagrams for reasoning flows - Charts for quantitative analysis - Formatted tables for comparative data ### Example Invocation ``` /think Why is the login endpoint returning 500 errors intermittently? ``` Creates analysis document with abstract-first structure, key findings with confidence levels, and visual reasoning chains. See `/think` command for full template. ## When to Use Quarto **Perfect for:** - LLM epistemological reasoning (use `/think` command) - Static reports and documentation (no interactivity needed) - Multi-format publishing (PDF + HTML + Word from single source) - Scientific documents (equations, citations, cross-references) - Presentations (RevealJS HTML slides, PowerPoint, Beamer PDF) - Websites and blogs (multi-page projects) - Exporting Jupyter notebooks for publication **NOT for:** - Interactive dashboards (use Shiny or dedicated dashboard tools) - Real-time data updates (use web dashboards) - When you need widgets/sliders for end users ## Data Provenance and Portability **CRITICAL: Quarto documents must be reproducible with documented dependencies.** When someone runs `quarto render analysis.qmd`, they should be able to get identical results given: - The document itself (`.qmd` file) - Documented external dependencies (with setup instructions) - Access to the same data sources (APIs, databases) ### The Portability Contract A `.qmd` file defines a **complete data pipeline**. The document contains: 1. **Data extraction logic** - How to obtain the data (queries, API calls, etc.) 2. **Transformation code** - How to process and analyze 3. **Presentation** - Charts, tables, narrative 4. **Dependency documentation** - Setup instructions for heavy dependencies **Practical limits**: Some dependencies are too expensive to rebuild on every render: - **Vector indexes** (LanceDB, FAISS) - Document how to build, reference existing - **Large datasets** - Commit to git or document extraction, don't re-download - **ML models** - Reference by path with setup instructions The key is **documentation**: readers must understand what's needed and how to set it up. ### Anti-Pattern: Opaque File References ❌ **BAD: Referencing local files without provenance** ```python df = pd.read_json('/tmp/orders.jsonl', lines=True) # Where did this come from? df = pd.read_csv('sales.csv') # Who created this? When? How? # "This JSONL file" without explaining its origin data = load_data('extracted_metrics.jsonl') # Non-portable! ``` ### Good Pattern: Embed Data Extraction ✅ **PREFERRED: Document defines where data comes from** ```python #| cache: true import pandas as pd from epq import bq # Extract from BigQuery via client library — cached to avoid re-running on every render chunks = list(bq.run_bq_query(""" SELECT * FROM production.orders WHERE date >= '2024-01-01' AND status = 'completed' """)) df = pd.concat(chunks, ignore_index=True) ``` ✅ **ALSO GOOD: Canonical external sources** ```python #| cache: true import pandas as pd # Public dataset with stable URL df = pd.read_csv('https://data.company.com/public/sales-2024.csv') # Or versioned data in the same repository df = pd.read_csv('data/sales-2024-v2.csv') # Committed to git with the .qmd ``` ### Heavy Dependencies: Document, Don't Rebuild **Rule of thumb**: Un-cached renders should complete in < 60 seconds. - **< 60 seconds** → Embed in document (with `cache: true`) - **> 60 seconds** → Document as external dependency with setup instructions Some dependencies are too expensive to recreate on every render. **Document them clearly** so readers can set up the environment. ✅ **GOOD: Reference with setup documentation** ```python #| echo: false import subprocess # DEPENDENCY: LanceDB index at ~/.lancedb/documents # Setup: lancer ingest -t documents ~/corpus/*.md # This index contains ~50k documents and takes ~10 min to build result = subprocess.run( ['lancer', 'search', '-t', 'documents', 'shipping rate errors', '--limit', '20'], capture_output=True, text=True, check=True ) relevant_docs = result.stdout ``` ✅ **GOOD: Prerequisites section in document** ```qmd --- title: "Knowledge Base Analysis" --- ## Prerequisites This analysis requires the following setup: 1. **LanceDB index**: `lancer ingest -t documents ~/corpus/*.md` 2. **BigQuery access**: Authenticated via `gcloud auth application-default login` 3. **Data snapshot**: Run `./scripts/extract-data.sh` (takes ~5 min) ## Analysis ... ``` ❌ **BAD: Silent dependency on local state** ```python # No documentation about what this index is or how to create it results = lancer.search("documents", "query") # Will fail for anyone else ``` ### Caching for Iteration Speed Use `cache: true` to avoid re-running expensive operations during iteration. **Requires jupyter-cache** (one-time install): ```bash uv add jupyter-cache ``` **Per-cell caching:** ```python #| cache: true #| label: data-extraction # This cell only re-executes if the code changes from epq import bq import pandas as pd chunks = list(bq.run_bq_query("SELECT ...")) df = pd.concat(chunks, ignore_index=True) ``` **Document-wide caching in YAML frontmatter:** ```yaml --- title: "Analysis Report" execute: cache: true --- ``` ### Freeze for Project-Level Caching For projects with many documents, use `freeze` to cache execution results in version control: ```yaml # _quarto.yml (project config) execute: freeze: auto # Re-render only when source changes ``` **Key difference:** - `cache: true` - Caches cell outputs locally (Jupyter Cache) - `freeze: auto` - Stores results in `_freeze/` directory (can commit to git) **When to use freeze:** - Large projects with many collaborators - Documents with environment-specific dependencies - When you want cached results portable across machines (commit `_freeze/`) ### Cache-Read-or-Query Pattern (BigQuery / Expensive APIs) Use this pattern when `cache: true` is insufficient — specifically when: - Querying BigQuery or other expensive external APIs - Cache must survive Quarto kernel restarts (Jupyter cache does not) - You want explicit control over cache invalidation (not tied to source changes) - Cache files are machine-specific and should be gitignored Set `execute: cache: false` in frontmatter when using this pattern (disable Jupyter cache to avoid double-caching). **Helper functions** (add once per document, in a setup cell): ```python import json from pathlib import Path from datetime import datetime, timezone _CACHE_DIR = Path('data/cache') _CACHE_DIR.mkdir(parents=True, exist_ok=True) def _cache_path(name): return _CACHE_DIR / f"{name}.json" def _read_cache(name): p = _cache_path(name) if p.exists(): return json.loads(p.read_text()) return None def _write_cache(name, records, scalars=None): p = _cache_path(name) p.write_text(json.dumps({ '_queried_at': datetime.now(timezone.utc).isoformat(), 'records': records, 'scalars': scalars or {} }, default=str)) ``` **Per-dataset usage template** (repeat for each dataset): ```python _c = _read_cache('my_dataset') if _c: df = pd.DataFrame(_c['records']) my_scalar = float(_c['scalars']['my_scalar']) else: df = run_bq_query(my_query) my_scalar = float(df['col'].values[0]) _write_cache('my_dataset', df.to_dict(orient='records'), { 'my_scalar': my_scalar, }) ``` **Cache hit** → reads DataFrame and scalars from JSON; no BigQuery call. **Cache miss** → queries BigQuery live, writes cache, continues render. **Query failure on miss** → render fails loudly (intentional — no silent fallback). **Never do:** - `except Exception: my_scalar = 42` — silent fallback masks broken queries - Assign a constant inside `try:` without a preceding query call — hidden constant, not a live value - Omit `_write_cache()` in the else branch — next render re-queries unnecessarily **Cache file format:** ```json { "_queried_at": "2026-02-20T16:00:00Z", "records": [...], "scalars": {...} } ``` Serialization notes: - Numpy arrays → store as lists (`df['col'].tolist()`), reconstruct with `np.array(...)` - Dates → use `default=str` in `json.dumps` to handle non-serializable types **Cache invalidation:** ``` # .gitignore data/cache/ # Justfile recipe delete-cache: rm -rf data/cache/ ``` ### Complete Example: Reproducible Analysis ```qmd --- title: "Q4 2024 Sales Analysis" author: "Josh Lane" date: "2024-12-31" format: gfm: wrap: none html: theme: dark: darkly light: flatly execute: cache: true filters: - auto-dark --- ## Data Extraction ```{python} #| cache: true #| label: extract-sales import subprocess import io import pandas as pd # Reproducible: Query is embedded in the document result = subprocess.run([ 'bigquery', 'query', ''' SELECT date, product, region, sales, units FROM production.sales WHERE EXTRACT(QUARTER FROM date) = 4 AND EXTRACT(YEAR FROM date) = …(truncated)

# Quarto > Render computational documents to markdown (DEFAULT), PDF, HTML, Word, and presentations using Quarto. PREFER markdown output for composability. Use for static reports, multi-format publishing, scientific documents with citations/cross-references, or exporting Jupyter notebooks. Triggers on "render markdown", "render PDF", "publish document", "create presentation", "quarto render", or multi-format publishing needs. - Skill: `lanej/quarto` (Agent Skill) - Install (CLI): `npx skillmds@latest add lanej/quarto` - Raw SKILL.md: https://api.skillmd.com/api/skills/lanej/quarto/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Web & Frontend - Author: lanej (https://skillmd.com/u/lanej) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/lanej/quarto --- # Quarto Skill Quarto is an open-source scientific and technical publishing system built on Pandoc. It renders computational documents (with Python, R, Julia code) to publication-quality output in multiple formats. ## Bootstrap with `epq scaffold` (PREFERRED for new projects) New QMD analysis projects must be created via the `epq` CLI — do NOT manually create pyproject.toml, _quarto.yml, justfile, figures/_style.py, or latex-header.tex: ```bash epq scaffold ~/workspace/projects/my-analysis # generates all boilerplate cd ~/workspace/projects/my-analysis just bootstrap # uv sync + ipykernel install epq audit . # verify clean (0 warnings) ``` `epq scaffold` generates: `_quarto.yml` (with `jupyter:` set), `latex-header.tex` (local copy — no external path dep at render time), `justfile` (thin import wrapper for canonical recipes), `pyproject.toml` (`package = false`, epq editable install), `.gitignore`, `figures/fig_example.py` (canonical dev loop), `{name}_files/figure-pdf/` pre-created. To audit or retrofit an existing project: ```bash epq audit # JSON violations with file/line/suggestion epq fix # unified diffs for auto-fixable issues (LLM reviews and applies) epq list-rules # enumerate all rule IDs ``` Shared library — import in every analysis QMD setup cell: ```python from epq import style, cache, bq, fmt style.apply_style() # canonical rcParams (150/200 DPI, sans-serif fallbacks) style.NAVY, style.TEAL, ... # workspace palette — never redefine inline cache.read_cache("name") # 24h TTL file-based cache → None if stale/missing bq.run_bq_query(SQL) # BigQuery client library wrapper → Iterator[DataFrame] fmt.millions_formatter() # FuncFormatter for ax.yaxis.set_major_formatter() ``` Full authoring reference: `~/src/analysis-doc/docs/AGENTS.md` Retrofit guide: `~/src/analysis-doc/docs/RETROFIT.md` --- ## External Python Figures Pattern (PREFERRED for complex documents) For documents with multiple visualizations, **extract all matplotlib code into standalone Python modules** in `figures/`. The QMD becomes a thin shell with stub cells only. ### Architecture ``` {name}.qmd ← thin shell: prose + data-load cells + stub figure cells only _quarto.yml ← jupyter: {name} (set by epq scaffold) latex-header.tex ← local copy (set by epq scaffold) justfile ← imports ~/src/analysis-doc/tools/justfile figures/ __init__.py fig_NAME.py ← one module per figure; render(data) contract scripts/data/ extract_NAME.py ← standalone BigQuery extractor; writes data/cache/*.json data/cache/ ← JSON cache files (gitignored) {name}_files/ figure-pdf/ ← dev loop writes here (pre-created by epq scaffold) ``` ### Figure Module Contract One file per figure (not a dispatcher). `render(data: dict)` is the only public function. ```python # figures/fig_revenue.py from pathlib import Path from epq import style, fmt # palette and formatters from epq — never local _style.py LABEL = "fig-revenue" # matches QMD #| label: fig-revenue FIG_WIDTH = 8.5 FIG_HEIGHT = 4.0 FIG_CAP = "Insight-focused caption — not a data description." def render(data: dict) -> None: """Render figure. Called from QMD stub cell with shared data dict. Do NOT call plt.show() or plt.savefig() here — Quarto handles capture. Do NOT call plt.close() here — handled in __main__ and between QMD cells. """ import matplotlib.pyplot as plt style.apply_style() fig, ax = plt.subplots(figsize=(FIG_WIDTH, FIG_HEIGHT)) # ... all matplotlib code, reading from data dict ... df = data.get("revenue", _load_sample_data()) ax.bar(df["month"], df["revenue"], color=style.NAVY) ax.yaxis.set_major_formatter(fmt.millions_formatter()) ax.set_title(FIG_CAP.rstrip(".")) plt.tight_layout() def _load_sample_data(): """Synthetic fallback for dev loop (no BQ needed).""" import pandas as pd return pd.DataFrame({"month": ["Q1", "Q2", "Q3", "Q4"], "revenue": [1.2e6, 1.4e6, 1.3e6, 1.6e6]}) if __name__ == "__main__": """Dev loop — saves to {project}_files/figure-pdf/{LABEL}-output-1.png. Writes to the same path Quarto uses, so visual inspection is against the real render artifact. Run via: just dev-fig revenue """ import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt _project_root = Path(__file__).parent.parent _out_dir = _project_root / f"{_project_root.name}_files" / "figure-pdf" _out_dir.mkdir(parents=True, exist_ok=True) out = str(_out_dir / f"{LABEL}-output-1.png") render({}) # synthetic data fallback plt.savefig(out, dpi=150, bbox_inches="tight") # savefig BEFORE close print(f"Saved {out}") plt.close("all") ``` ### QMD Stub Cell ```python #| label: fig-revenue #| fig-cap: "Revenue by stream." #| fig-height: 4.0 #| fig-width: 8.5 #| fig-pos: "H" #| out-width: 100% import sys; sys.path.insert(0, str(Path("."))) if "." not in sys.path else None from figures import fig_revenue fig_revenue.render(data) ``` ### Justfile (generated by `epq scaffold`) Projects get a thin justfile that imports the canonical recipe library: ```just # Project-local overrides go here. Canonical recipes imported below. import? '~/src/analysis-doc/tools/justfile' ``` The canonical justfile provides: ```just # Render figure to {project}_files/figure-pdf/fig-NAME-output-1.png dev-fig NAME: PYTHONPATH=. uv run python figures/fig_{{NAME}}.py @echo "→ $(basename $PWD)_files/figure-pdf/fig-{{NAME}}-output-1.png" # Full render (clears Jupyter cache) render: rm -rf .jupyter_cache/ quarto render *.qmd ``` ### Figure Audit and Visual Iteration Protocol **CRITICAL: Any task involving figures — iteration, audit, or review — MUST render and visually inspect the PNG before reporting. Code-only review is incomplete.** **Required sequence for every figure change or audit:** 1. `just dev-fig NAME` → `{project}_files/figure-pdf/fig-NAME-output-1.png` 2. **Read the PNG using the Read tool** — do not skip this step 3. Apply the visual readability checklist to what you see in the PNG 4. Fix issues, re-render, re-inspect until checklist passes 5. Only then check factual accuracy against prose/data **Do NOT use `just preview-fig` / Playwright** — it screenshots HTML chrome, not the raw figure. Read the PNG directly from `{project}_files/figure-pdf/`. **Visual Readability Checklist (inspect the rendered PNG, not the code):** - [ ] **Suptitle not clipped** — `suptitle(y=1.02)` ALWAYS clips in PDF. Use `y=1.0` + `fig.subplots_adjust(top=0.82–0.88)`. Never use `y > 1.0`. - [ ] **No text collisions** — suptitle and panel titles have clear separation - [ ] **Text readable at print scale** — PDF page is 6.5in wide; >1.5× zoom = illegible - [ ] **Panels not squished** — 3-panel: FIG_HEIGHT ≥ 4.0; single-panel ≥ 3.0; swim-lane ≥ 4.5 - [ ] **Bar labels within bounds** — `set_clip_on(True)` makes labels invisible, not small - [ ] **No partial annotations** — text near ylim/xlim edges fully in frame - [ ] **Text contrast correct** — see color guard below **Text contrast rule (most common source of invisible text):** ```python # Use epq.style helper — covers all palette fills correctly: from epq import style tc = style.text_color_for(fill_color) # WHITE for dark fills, NAVY for light # Manual guard if needed: DARK_FILLS = (style.NAVY, style.TEAL, style.CORAL, style.PURPLE, style.GOLD) tc = style.WHITE if fill in DARK_FILLS else style.NAVY # ^^^^ NAVY, never SLATE # SLATE on LIGHT_SLATE = 2.6:1 contrast — fails AA, looks muddy at print scale ``` Verified contrast ratios: - NAVY/TEAL/CORAL/PURPLE/GOLD fills → WHITE text (8–16:1 ✅) - LIGHT_SLATE fill → NAVY text (5.5:1 ✅); SLATE fails (2.6:1 ❌) - Pale backgrounds (LIGHT_BG, NAVY_BG) → NAVY text; WHITE is invisible (~1.07:1 ❌) **Common visual defects invisible in code:** - Suptitle bleeding into panel titles → increase FIG_HEIGHT, use `subplots_adjust` - Bar labels extending past xlim → invisible (not small); expand xlim or reduce offset - Blank/white PNG → `plt.savefig()` called after `plt.close()`. Fix: call savefig first: ```python render({}) plt.savefig(out, dpi=150, bbox_inches="tight") plt.close("all") ``` ### Key Rules - **Edit `figures/fig_NAME.py`, not the QMD** — QMD stubs never change unless label/caption/dimensions change - **One module per figure** — `render(data: dict) -> None`, no dispatcher pattern - **`from epq import style, fmt`** — never a local `_style.py` copy - **`PYTHONPATH=.`** required when running modules directly: `PYTHONPATH=. uv run python figures/fig_NAME.py` - **`render()` must NOT call `plt.show()`, `plt.close()`, or `plt.savefig()`** — Quarto handles capture; `__main__` handles save - **`matplotlib.use("Agg")`** before any pyplot import in `__main__` - **`data` dict** is the only input — never read cache files inside figure modules; provide synthetic fallback in `_load_sample_data()` - **Dev loop output**: `{project}_files/figure-pdf/{LABEL}-output-1.png` (pre-created by `epq scaffold`) - **Reference implementation**: `~/workspace/projects/luma-revenue-forecast/` and `~/workspace/projects/revenue-forecast-2026/` --- ## PDF Project Bootstrap Use `epq scaffold` — it handles all of the following automatically: ```bash epq scaffold ~/workspace/projects/my-project cd ~/workspace/projects/my-project just bootstrap # runs: uv sync + ipykernel install + mkdir data/cache ``` Manual checklist (only if NOT using `epq scaffold`): 1. Create `pyproject.toml` with `[tool.uv] package = false` and run `uv sync` 2. Register kernel: `uv run python -m ipykernel install --user --name=` 3. `_quarto.yml`: set `jupyter: ` (must match registered kernel name exactly) 4. Copy `latex-header.tex` locally from `~/src/analysis-doc/templates/` (do not reference it via external path) 5. Never borrow another project's venv or kernel — each project must own its own ## PDF Title Suppression - **Never use YAML `title:` or `subtitle:`** — they produce `\maketitle` which double-renders with any custom header - Use a raw LaTeX inline header in the document body instead: ```{=latex} {\large\textbf{Document Title}} \hfill {\small\color{gray} Author \quad\textbullet\quad \today} \vspace{4pt} \hrule \vspace{8pt} ``` - Add `pagetitle: " "` to YAML to suppress the HTML `` without breaking Pandoc ## PDF Figure Rules ### Split multi-story panels If a combined figure has two sub-panels telling different insights, split into separate `fig-*` cells with their own captions. Ask upfront whether panels should be split — this is cheaper than rework after the fact. ### Legend placement - Top-right (`loc='upper right'`) when data occupies the bottom portion of the chart - Below chart when dense: `fig.legend(loc='lower center', ncol=N, bbox_to_anchor=(0.5, -0.02))` + `fig.subplots_adjust(bottom=0.20)` - Never place legend over data area ### Date axes — always explicit Never rely on `AutoDateLocator` — it overcrowds on multi-year spans: ```python ax.xaxis.set_major_locator(mdates.MonthLocator(bymonth=[1, 7])) ax.xaxis.set_major_formatter(mdates.DateFormatter("%b '%y")) ax.tick_params(axis='x', labelsize=8) ``` ### Never auto-open artifacts Justfile render recipes must not include `&& open <file>`. The user opens files manually. ## LaTeX Max-Runs Warning "WARN: maximum number of runs (9) reached" is **cosmetically harmless** when there are no `\ref{}` cross-references in prose. It is caused by `fancyhdr` + many figures creating layout oscillation. Mitigations: - Use `\needspace` not `\clearpage` (see needspace sizing table in the `\needspace` section) - Remove `labelformat=empty` from `\captionsetup` — caption numbering churn drives oscillation - Use `htbp` float placement rather than forced `H` `\needspace` sizing reference — X = figure height + 1.2in overhead: | Figure height | `\needspace` | |---|---| | 3.2in | `\needspace{4.5in}` | | 4.0in | `\needspace{5.2in}` | | 5.5in | `\needspace{6.8in}` | | Prose only | `\needspace{2.5in}` | ## BigQuery + Pandas Gotchas (in Quarto Documents) - BigQuery returns nullable `Int64` for integer columns — always `.astype('float64')` before `fillna()` - Quarterly data on a monthly x-axis: `df.set_index('quarter').reindex(monthly_idx, method='ffill')` + `ax.step(..., where='post')` - Never put `\n` inside BigQuery SQL string literals in Python f-strings — use spaces instead - Always define intermediate variables BEFORE the `Markdown(f"""...""")` call — f-strings evaluate at call time, not definition time --- ## Best Practices (TL;DR) 1. **Markdown-First**: Default to `format: gfm` with `wrap: none` for composability, portability, and archival 2. **No Line Wrapping**: Always use `wrap: none` for GFM output (avoids artificial line breaks) 3. **Dark Mode**: Always use `auto-dark` filter with dual themes for HTML output (accessibility and modern UX) 4. **Visual Expression**: Use charts and formatted tables, NEVER raw data dumps (`df.head()`, `print(dict)`) 5. **LaTeX for Math**: Use LaTeX notation for ALL mathematical expressions ($\alpha = 0.15$, not "alpha = 0.15") 6. **Professional Tables**: Use LaTeX tables (booktabs) for PDF, Great Tables for HTML 7. **No TOC**: Table of contents is usually noise - use clear section headings instead 8. **PDF for Sharing**: Use `--to pdf` for Google Drive sharing (read-only, professional) 9. **Blank Lines Before Lists**: ALWAYS include a blank line before every list (bullet or numbered) - no exceptions 10. **No Appendix for Sources**: Data sources belong in code blocks, not appendix - only add external sources not directly referenced in code to appendix 11. **Use Markdown() Class**: ALWAYS use `Markdown()` for text output in code blocks - NEVER use `print()` or `printf()` (output must render as formatted markdown) 12. **Minimal PDF Titling**: For PDF output, suppress YAML `title`/`author`/`date` fields (they produce an academic title block via `\maketitle`). Use a raw LaTeX minipage inline header instead. Use `##` markdown headings for section headings (NOT raw LaTeX `\noindent{\large\textbf{...}}` blocks — those fight with Quarto's float placement). Exception: only use raw LaTeX headings for the document title line itself. 13. **PDF Figure Sizing**: Every chart chunk MUST have `#| fig-pos: "H"`, `#| fig-width: N`, `#| fig-height: N`, `#| out-width: 100%`. Always end chunks with `plt.close('all')`. Set `ax.text(...).set_clip_on(True)` on all annotation labels. Never use mixed coordinate transforms. See "PDF Figure Sizing — Critical Patterns" section for full details. ## PDF Figure Sizing — Critical Patterns **CRITICAL: Matplotlib figure sizing in Quarto PDF output is failure-prone. Follow these rules exactly.** ### The Working Pattern (copy verbatim) Every chart chunk that renders to PDF must have ALL of these chunk options: ```python #| label: fig-my-chart #| fig-pos: "H" # force-here via float.sty — prevents deferral/stacking #| fig-width: 6.5 # must match figsize width in Python code #| fig-height: 3.5 # must match figsize height in Python code #| out-width: 100% # tells LaTeX to scale to full text column width #| fig-cap: "Caption text with no bare % characters — write 'percent' instead." ``` And in the Python code: ```python fig, ax = plt.subplots(figsize=(6.5, 3.5)) # must match chunk fig-width/fig-height # ... chart code ... plt.tight_layout() plt.show() plt.close('all') # REQUIRED — prevents state leaking between chunks ``` ### rcParams That Must Not Be Changed ```python plt.rcParams.update({ 'savefig.bbox': None, # fills declared figsize exactly — do NOT set to 'tight' 'savefig.pad_inches': 0, 'figure.dpi': 200, 'savefig.dpi': 300, }) ``` **`savefig.bbox: None`** is counterintuitive but correct for PDF output. Setting it to `'tight'` causes matplotlib to auto-expand the canvas, which fights against the declared figsize and produces malformed figure PDFs. ### Root Causes of "Comically Small" Figures These are the diagnosed failure modes, in order of frequency: **1. Text labels extending beyond xlim/ylim — MOST COMMON** ```python # ❌ BROKEN: annotation text positioned beyond axis limits for bar, row in zip(bars, df.itertuples()): ax.text(bar.get_width() + 4, ..., f"{row.value}") # +4 may push past xlim ax.set_xlim(0, 380) # text at bar.width+4 can exceed 380 → bbox explosion # ✅ FIX: clip text labels to axes, OR increase xlim to accommodate labels for bar, row in zip(bars, df.itertuples()): t = ax.text(bar.get_width() + 4, ..., f"{row.value}") t.set_clip_on(True) # ← prevents bbox from expanding to include clipped text # OR: ax.set_xlim(0, 420) # ensure xlim accommodates largest label ``` When `ax.text()` labels are placed at `bar.get_width() + offset` in data coordinates and those labels extend beyond `xlim`, the PDF backend measures the full artist bounding box (including out-of-bounds text) when computing the figure's page size. This causes the figure PDF to be output at **half or less of the declared figsize** — which LaTeX then renders at postage-stamp size even though `out-width: 100%` is set. **Rule:** After setting `xlim`, add `t.set_clip_on(True)` to ALL `ax.text()` calls, or add enough xlim headroom to fit the longest annotation. **2. Mixed coordinate transforms on annotations** ```python # ❌ BROKEN: mixes data coordinates with axis-fraction transform ax.annotate("", xy=(x0, y0 + 3), xytext=(x1, y1 + 3), ...) # data coords ax.text(0.5, max_val + 9, "label", transform=ax.get_xaxis_transform()) # mixed! # ✅ FIX: use pure axes fraction for floating annotations ax.annotate("label text", xy=(0.5, 0.85), xycoords='axes fraction', ha='center', va='center', fontsize=9, ...) ``` `ax.get_xaxis_transform()` mixes x=axis-fraction with y=data coordinates. When the y-value in data coords exceeds ylim, the PDF backend's bounding box measurement goes pathological, producing figures that are 10-30× taller than declared. **Always use pure coordinate systems** — either all data coords or all `xycoords='axes fraction'`. **3. Missing `plt.close('all')` between chunks** Without `plt.close('all')` after each `plt.show()`, matplotlib figure state (transforms, layout engines, bounding boxes) leaks between Jupyter/Quarto execution chunks. This can cause later charts to inherit corrupt layout state from earlier ones. Always end every chart chunk with: ```python plt.tight_layout() plt.show() plt.close('all') ``` **4. `fig-pos: "!ht"` instead of `"H"`** `"!ht"` (try-here, then top-of-page) causes LaTeX to defer figures when there's insufficient space, stacking them at awkward positions. `"H"` (force-here via `float.sty`) places the figure exactly where declared. Requires `\usepackage{float}` in the LaTeX header. **5. Missing `#| fig-width` / `#| fig-height` on chunk** Without explicit chunk-level sizing, Quarto uses YAML defaults and may not pre-allocate the correct float box size before Python renders into it. Always specify both per-chunk. ### Diagnosing Figure Size Issues To identify which figures are malformed without waiting for a full visual review: ```bash # Add keep-tex to render temporarily cd /path/to/doc && uv run quarto render doc.qmd --to pdf -M keep-tex:true # Check actual page dimensions of each generated figure PDF for f in doc_files/figure-pdf/*.pdf; do echo -n "$f: " pdfinfo "$f" 2>/dev/null | grep "Page size" done # A correct 6.5×3.5in figure should be ~468×252 pts (at 72 pts/in) # A figure with width << 440 pts or height >> 400 pts is malformed ``` ### The Nuclear Option If a figure keeps rendering incorrectly despite all fixes, force PNG raster output: ```python #| label: fig-problematic #| fig-pos: "H" #| dev: png # ← bypass the PDF vector pipeline entirely #| dpi: 150 #| fig-width: 6.5 #| fig-height: 3.5 #| out-width: 100% ``` PNG output is immune to all the bbox/transform issues because matplotlib renders to a fixed-size raster and Quarto embeds it directly. Use as a last resort since vector PDF is crisper. ### Caption Numbering ```latex % Restore default numbering (Figure 1., Figure 2., etc.) with bold prefix: \captionsetup{font={small,it},justification=centering,skip=6pt,labelfont=bf} % Suppress numbering (caption text only, no "Figure N." prefix): \captionsetup{font={small,it},justification=centering,skip=6pt,labelformat=empty,labelsep=none} ``` **Never use bare `%` in `fig-cap` strings** — write "percent" or "percentage points" instead. LaTeX may fail to compile depending on pandoc version. ### Reference Lines (axvline/axhline) Opacity Reference lines (baselines, averages) should be visible context, not dominant elements. Always set `alpha=0.4`: ```python ax.axvline(48.9, color=NAVY, linestyle=":", linewidth=1.6, alpha=0.4, label="Baseline", zorder=3) ax.axhline(37.8, color=SLATE, linestyle="--", linewidth=1.4, alpha=0.4, label="Avg", zorder=3) ``` At `alpha=1.0` (default), reference lines dominate the chart and compete with the data bars. `alpha=0.4` keeps them readable without visual dominance. --- ## Visual Expression Philosophy **CRITICAL: Quarto documents are for COMMUNICATION, not raw data dumps.** Quarto outputs are static documents meant to convey insights to humans. Raw dataframes, print statements, and JSON blobs fail to communicate effectively. ### Visual Hierarchy (Use in Order) 1. **Charts/Plots** - For trends, distributions, comparisons, relationships 2. **Formatted Tables** - For structured data with styling and context 3. **Formatted Metrics** - For key numbers with context and formatting 4. **Raw Output** - NEVER (not even for debugging - use separate analysis files) ### Anti-Patterns: What NOT to Do ```python # ❌ BAD: Raw dataframe dump df.head() # ❌ BAD: Print statements - output renders as plain text, not formatted markdown print(f"Total: {total}") print(data_dict) # ❌ BAD: printf/print for text - use Markdown() instead print("## Summary\n- Item 1\n- Item 2") # Renders as plain text! # ❌ BAD: Bare variable returning data structure result # Returns raw dict/JSON # ❌ BAD: DataFrame info without formatting df.describe() df.info() # ✅ GOOD: Use Markdown() for ALL text output from IPython.display import Markdown Markdown(f""" ## Summary - **Total**: {total:,} - **Average**: {avg:.2f} """) ``` ```markdown ❌ BAD: Plain text for mathematical notation - The growth rate is alpha = 0.15 or 15% - We calculated the mean mu = sum(xi)/n - The correlation coefficient r = 0.85 ❌ BAD: No table formatting ```python print(df.head()) ``` ❌ BAD: Using asterisks for equations - E = m * c^2 - y = beta0 + beta1 * x ``` ### Good Patterns: Visual Communication ```python # ✅ GOOD: Chart for trends import matplotlib.pyplot as plt fig, ax = plt.subplots(figsize=(10, 6)) df.groupby('date')['sales'].sum().plot(ax=ax, kind='line') ax.set_title('Sales Trend Over Time') ax.set_ylabel('Sales ($)') plt.tight_layout() plt.show() # ✅ GOOD: Formatted table using Great Tables from great_tables import GT (GT(df.head(10)) .tab_header(title="Top 10 Sales Records") .fmt_currency(columns="sales", currency="USD") .fmt_date(columns="date", date_style="medium")) # ✅ GOOD: Formatted table using pandas markdown from IPython.display import Markdown Markdown(df.head(10).to_markdown(index=False, tablefmt='pipe')) # ✅ GOOD: Formatted metrics in markdown with LaTeX from IPython.display import Markdown Markdown(f""" ## Key Metrics - **Total Sales**: ${total_sales:,.2f} - **Average Order**: ${avg_order:,.2f} - **Growth Rate**: $\\alpha = {growth_rate:.1%}$ (15% YoY) - **Top Product**: {top_product} ### Statistical Summary The linear regression model $y = \\beta_0 + \\beta_1 x + \\epsilon$ yielded: - Slope: $\\hat{{\\beta_1}} = 3.2$ (SE = 0.4) - $R^2 = 0.78$, indicating strong fit """) # NOTE: Mermaid diagrams must use native Quarto syntax outside Python blocks # Use ```{mermaid} directly in markdown, NOT inside Markdown() calls ``` ### Why Visual Expression Matters - **Documents are for humans**: Show insights, not data structures - **Static format**: No interactive exploration - must communicate clearly on first view - **Traceable reasoning**: Visualize fact→conclusion chains, not raw JSON - **Professional output**: Charts and tables look polished in PDF/HTML/Word - **Accessibility**: Visual hierarchy helps readers navigate content - **Shareability**: Well-formatted outputs communicate without explanation ## Narrative Structure **Build understanding progressively through a series of sections:** ### Document Flow 1. **Abstract** - The punchline first (executive summary for busy readers) 2. **Key Findings** - Scannable bullet points with confidence levels 3. **Base Facts** - Individual observations/data, each in its own section 4. **Synthesis Sections** - Combine earlier facts into higher-level insights 5. **Conclusion** - Final synthesis referencing the insights above ### Base Facts (Individual Sections) Each base fact is an independent observation with its own data and evidence. Use descriptive headers (not "Fact 1"): ```qmd ## Response Time Distribution \needspace{3in} Analysis of the past 7 days shows significant tail latency: - p50: 45ms - p95: 230ms - p99: 890ms (concerning) ```{python} #| echo: false # Chart showing latency distribution ``` ``` ### Synthesis Sections Synthesis sections **explicitly reference** which earlier sections they build upon: ```qmd ## Performance Degradation Under Load \needspace{4in} Building on the response time distribution and traffic patterns above, we observe a clear correlation: p99 latency spikes to 2.3s during the 2-4pm peak traffic window. The system handles baseline load well but degrades significantly under peak conditions. ``` ### Keeping Content Together (PDF) Use `\needspace{Xin}` before sections with charts/diagrams to prevent awkward page breaks and large whitespace gaps: ```qmd ## Revenue by Carrier \needspace{4in} ```{python} # Chart code here ``` ``` **Guidelines:** - **Mermaid diagrams**: `\needspace{2in}` - **Single chart**: `\needspace{3in}` - **Chart + explanation**: `\needspace{4in}` **Requires** in YAML frontmatter: ```yaml format: pdf: include-in-header: text: | \usepackage{needspace} ``` ## Markdown Formatting Rules (CRITICAL) **IMPORTANT: Markdown requires blank lines between paragraphs for proper rendering.** ### Line Breaks and Paragraphs **The Problem:** ```markdown ❌ BAD: This will render as one long line This text appears on a new line in the source But it renders on the same line as above Because there's no blank line between them ``` **The Solution:** ```markdown ✅ GOOD: This renders as separate paragraphs This text appears on its own line because there's a blank line above it. Each paragraph needs a blank line before and after it. ``` ### Common Markdown Patterns **Paragraphs (need blank lines):** ```markdown This is paragraph one. This is paragraph two. This is paragraph three. ``` **Lists REQUIRE blank line before the list:** ```markdown ❌ BAD: List doesn't render correctly Here is some text: - Item 1 - Item 2 ✅ GOOD: Blank line before list Here is some text: - Item 1 - Item 2 ``` **Numbered lists also require blank line before:** ```markdown ❌ BAD: Numbered list broken The steps are: 1. First step 2. Second step ✅ GOOD: Blank line before numbered list The steps are: 1. First step 2. Second step ``` **Lists (no blank lines between items):** ```markdown - Item 1 - Item 2 - Item 3 ``` **Multi-paragraph list items (blank lines within item):** ```markdown - Item 1 with first paragraph Item 1 continued with second paragraph (indented 2 spaces) - Item 2 starts here ``` **Headers (blank line before and after):** ```markdown Previous paragraph ends here. ## Section Header New paragraph starts here. ``` **Code blocks (blank line before and after):** ````markdown Previous paragraph ends here. ```python print("code block") ``` New paragraph starts here. ```` ### Quarto-Specific Markdown **Python code output:** ```python #| echo: false from IPython.display import Markdown # ❌ BAD: Single newline won't create paragraph break Markdown("Line 1\nLine 2") # Renders as: Line 1 Line 2 # ✅ GOOD: Double newline creates paragraph break Markdown("Line 1\n\nLine 2") # Renders as separate paragraphs # ✅ GOOD: Use triple-quoted string with blank lines Markdown(""" Paragraph one. Paragraph two. Paragraph three. """) ``` **F-strings in Markdown:** ```python #| echo: false from IPython.display import Markdown # ✅ GOOD: Blank lines between paragraphs Markdown(f""" ## Analysis Results The growth rate is {growth_rate:.1%}. This represents a significant increase over last quarter. We recommend increasing inventory by {inventory_increase:,} units. """) ``` ### Best Practices ✅ **DO:** - Use blank lines between all paragraphs - Use blank lines before and after headers - Use blank lines before and after code blocks - Use blank lines before and after tables - Use triple-quoted strings for multi-line markdown in Python ❌ **DON'T:** - Use single newlines and expect paragraph breaks - Forget blank lines around headers or code blocks - Mix single and double newlines inconsistently - Use `\n` in strings expecting paragraph breaks (use `\n\n`) ### Testing Markdown Formatting **Quick test:** ```bash # Create test document cat > test.qmd << 'EOF' --- title: "Markdown Test" format: gfm --- Paragraph 1. Paragraph 2. ## Header Paragraph 3. EOF # Render and check output quarto render test.qmd --to gfm cat test.md ``` ## LLM Self-Reasoning with Quarto Use `/think` command for structured analysis with graduated detail. ### Document Structure Documents use graduated detail so readers can stop at their desired depth: **Abstract** (paragraph) - Self-contained executive summary - Complete story: what, why, result, meaning - Include key metrics and confidence assessment **Key Findings** (3-5 bullets) - Result + confidence + brief evidence - Scannable - each finding valuable standalone - Format: `**[Finding]**: [Result] — [Evidence] (Confidence)` **Investigation** (detailed) - Observations (sourced facts with academic citations `[source]`) - Analysis (visual reasoning, statistical evidence) - Interpretation (what it means, confidence, dependencies) **Appendix** (optional) - Investigation notes, dead ends, debugging traces - External data sources NOT directly referenced in code blocks (e.g., verbal conversations, meeting notes, prior analyses) - Do NOT duplicate data sources already expressed in code blocks - the code IS the source documentation ### Visual Evidence Include diagram, chart, or table for most findings: - Mermaid diagrams for reasoning flows - Charts for quantitative analysis - Formatted tables for comparative data ### Example Invocation ``` /think Why is the login endpoint returning 500 errors intermittently? ``` Creates analysis document with abstract-first structure, key findings with confidence levels, and visual reasoning chains. See `/think` command for full template. ## When to Use Quarto **Perfect for:** - LLM epistemological reasoning (use `/think` command) - Static reports and documentation (no interactivity needed) - Multi-format publishing (PDF + HTML + Word from single source) - Scientific documents (equations, citations, cross-references) - Presentations (RevealJS HTML slides, PowerPoint, Beamer PDF) - Websites and blogs (multi-page projects) - Exporting Jupyter notebooks for publication **NOT for:** - Interactive dashboards (use Shiny or dedicated dashboard tools) - Real-time data updates (use web dashboards) - When you need widgets/sliders for end users ## Data Provenance and Portability **CRITICAL: Quarto documents must be reproducible with documented dependencies.** When someone runs `quarto render analysis.qmd`, they should be able to get identical results given: - The document itself (`.qmd` file) - Documented external dependencies (with setup instructions) - Access to the same data sources (APIs, databases) ### The Portability Contract A `.qmd` file defines a **complete data pipeline**. The document contains: 1. **Data extraction logic** - How to obtain the data (queries, API calls, etc.) 2. **Transformation code** - How to process and analyze 3. **Presentation** - Charts, tables, narrative 4. **Dependency documentation** - Setup instructions for heavy dependencies **Practical limits**: Some dependencies are too expensive to rebuild on every render: - **Vector indexes** (LanceDB, FAISS) - Document how to build, reference existing - **Large datasets** - Commit to git or document extraction, don't re-download - **ML models** - Reference by path with setup instructions The key is **documentation**: readers must understand what's needed and how to set it up. ### Anti-Pattern: Opaque File References ❌ **BAD: Referencing local files without provenance** ```python df = pd.read_json('/tmp/orders.jsonl', lines=True) # Where did this come from? df = pd.read_csv('sales.csv') # Who created this? When? How? # "This JSONL file" without explaining its origin data = load_data('extracted_metrics.jsonl') # Non-portable! ``` ### Good Pattern: Embed Data Extraction ✅ **PREFERRED: Document defines where data comes from** ```python #| cache: true import pandas as pd from epq import bq # Extract from BigQuery via client library — cached to avoid re-running on every render chunks = list(bq.run_bq_query(""" SELECT * FROM production.orders WHERE date >= '2024-01-01' AND status = 'completed' """)) df = pd.concat(chunks, ignore_index=True) ``` ✅ **ALSO GOOD: Canonical external sources** ```python #| cache: true import pandas as pd # Public dataset with stable URL df = pd.read_csv('https://data.company.com/public/sales-2024.csv') # Or versioned data in the same repository df = pd.read_csv('data/sales-2024-v2.csv') # Committed to git with the .qmd ``` ### Heavy Dependencies: Document, Don't Rebuild **Rule of thumb**: Un-cached renders should complete in < 60 seconds. - **< 60 seconds** → Embed in document (with `cache: true`) - **> 60 seconds** → Document as external dependency with setup instructions Some dependencies are too expensive to recreate on every render. **Document them clearly** so readers can set up the environment. ✅ **GOOD: Reference with setup documentation** ```python #| echo: false import subprocess # DEPENDENCY: LanceDB index at ~/.lancedb/documents # Setup: lancer ingest -t documents ~/corpus/*.md # This index contains ~50k documents and takes ~10 min to build result = subprocess.run( ['lancer', 'search', '-t', 'documents', 'shipping rate errors', '--limit', '20'], capture_output=True, text=True, check=True ) relevant_docs = result.stdout ``` ✅ **GOOD: Prerequisites section in document** ```qmd --- title: "Knowledge Base Analysis" --- ## Prerequisites This analysis requires the following setup: 1. **LanceDB index**: `lancer ingest -t documents ~/corpus/*.md` 2. **BigQuery access**: Authenticated via `gcloud auth application-default login` 3. **Data snapshot**: Run `./scripts/extract-data.sh` (takes ~5 min) ## Analysis ... ``` ❌ **BAD: Silent dependency on local state** ```python # No documentation about what this index is or how to create it results = lancer.search("documents", "query") # Will fail for anyone else ``` ### Caching for Iteration Speed Use `cache: true` to avoid re-running expensive operations during iteration. **Requires jupyter-cache** (one-time install): ```bash uv add jupyter-cache ``` **Per-cell caching:** ```python #| cache: true #| label: data-extraction # This cell only re-executes if the code changes from epq import bq import pandas as pd chunks = list(bq.run_bq_query("SELECT ...")) df = pd.concat(chunks, ignore_index=True) ``` **Document-wide caching in YAML frontmatter:** ```yaml --- title: "Analysis Report" execute: cache: true --- ``` ### Freeze for Project-Level Caching For projects with many documents, use `freeze` to cache execution results in version control: ```yaml # _quarto.yml (project config) execute: freeze: auto # Re-render only when source changes ``` **Key difference:** - `cache: true` - Caches cell outputs locally (Jupyter Cache) - `freeze: auto` - Stores results in `_freeze/` directory (can commit to git) **When to use freeze:** - Large projects with many collaborators - Documents with environment-specific dependencies - When you want cached results portable across machines (commit `_freeze/`) ### Cache-Read-or-Query Pattern (BigQuery / Expensive APIs) Use this pattern when `cache: true` is insufficient — specifically when: - Querying BigQuery or other expensive external APIs - Cache must survive Quarto kernel restarts (Jupyter cache does not) - You want explicit control over cache invalidation (not tied to source changes) - Cache files are machine-specific and should be gitignored Set `execute: cache: false` in frontmatter when using this pattern (disable Jupyter cache to avoid double-caching). **Helper functions** (add once per document, in a setup cell): ```python import json from pathlib import Path from datetime import datetime, timezone _CACHE_DIR = Path('data/cache') _CACHE_DIR.mkdir(parents=True, exist_ok=True) def _cache_path(name): return _CACHE_DIR / f"{name}.json" def _read_cache(name): p = _cache_path(name) if p.exists(): return json.loads(p.read_text()) return None def _write_cache(name, records, scalars=None): p = _cache_path(name) p.write_text(json.dumps({ '_queried_at': datetime.now(timezone.utc).isoformat(), 'records': records, 'scalars': scalars or {} }, default=str)) ``` **Per-dataset usage template** (repeat for each dataset): ```python _c = _read_cache('my_dataset') if _c: df = pd.DataFrame(_c['records']) my_scalar = float(_c['scalars']['my_scalar']) else: df = run_bq_query(my_query) my_scalar = float(df['col'].values[0]) _write_cache('my_dataset', df.to_dict(orient='records'), { 'my_scalar': my_scalar, }) ``` **Cache hit** → reads DataFrame and scalars from JSON; no BigQuery call. **Cache miss** → queries BigQuery live, writes cache, continues render. **Query failure on miss** → render fails loudly (intentional — no silent fallback). **Never do:** - `except Exception: my_scalar = 42` — silent fallback masks broken queries - Assign a constant inside `try:` without a preceding query call — hidden constant, not a live value - Omit `_write_cache()` in the else branch — next render re-queries unnecessarily **Cache file format:** ```json { "_queried_at": "2026-02-20T16:00:00Z", "records": [...], "scalars": {...} } ``` Serialization notes: - Numpy arrays → store as lists (`df['col'].tolist()`), reconstruct with `np.array(...)` - Dates → use `default=str` in `json.dumps` to handle non-serializable types **Cache invalidation:** ``` # .gitignore data/cache/ # Justfile recipe delete-cache: rm -rf data/cache/ ``` ### Complete Example: Reproducible Analysis ```qmd --- title: "Q4 2024 Sales Analysis" author: "Josh Lane" date: "2024-12-31" format: gfm: wrap: none html: theme: dark: darkly light: flatly execute: cache: true filters: - auto-dark --- ## Data Extraction ```{python} #| cache: true #| label: extract-sales import subprocess import io import pandas as pd # Reproducible: Query is embedded in the document result = subprocess.run([ 'bigquery', 'query', ''' SELECT date, product, region, sales, units FROM production.sales WHERE EXTRACT(QUARTER FROM date) = 4 AND EXTRACT(YEAR FROM date) = …(truncated)