Project Site Generator
Create and build a static HTML site for a research project.
Status of the canonical implementation
The rendering machinery lives in the sitekit package under
research-kit/sitekit/. Each project's source/site/build_all.py is a
thin shim that hands a SiteConfig to sitekit.build_site().
Migrated to sitekit: audit, campaign-finance, deterrence,
electoral-justice, fisc, judgeGPT, poll-sponsor-bias, promotor,
serasa, vague, lawsuit, scheme, segredo, connect (minimal +
empirical), and bind (theoretical). For migrated projects, verification is
content-parity of each rendered doc body against the pre-migration output plus
intended convergence of the chrome onto the shared design system — NOT
byte-equivalence (that only held for serasa, whose templates were the
extraction source; other projects had drifted chrome the migration deliberately
normalizes). The standard acceptance check the recent migrations used: page-set
diff (0 lost), body-text parity (~1.0; sub-1.0 is pure CSS-var normalization),
and "new broken-link set ⊆ baseline broken-link set" (the migration introduces
no broken links; pre-existing stale source refs stay pre-existing).
Not yet migrated — each is a DEDICATED effort blocked on a specific host or build, NOT a batch slot (verified by attempted migrations 2026-08-31, which correctly refused to force a lossy/unverifiable rewrite of coauthor-facing sites):
procure— coauthor-facing + validation-gated; ~2,500 loc of bespoke subsystems to port as hooks: a paper page with validation-ledger UI + number-attribution tooltips + feedback-worker (the blocker — 850 loc of bespoke UI that can't be built/verified without a TeX host), a dataset portal (build/summary/*.json, different schema than the empirical archetype), a 1,111-locbuild_cases.pycases renderer, validation + election-RD pages, and a chip-filtered AN index. ~40% maps to stock. Needs a TeX-capable host. Deploys plaintext-public → run a PII/sensitive compliance audit before any redeploy (sitekit build newly copiesbuild/figure+build/analysisPNGs and stops shippingmain.pdf).ficha— empirical, 476 pages (231 script pages dominate), coauthor-facing. Needs a TeX host: its bespoke paper page (inline-footnote reconstruction,\newcommandmacro substitution, side-rail TOC, figure-path rewrite) has no cachedbuild/make4ht/paper.htmlin-sandbox, so it can't be diffed — don't port it blind. Also renders everything FLAT at site root with a bespoke autolink layer (an_file_map,link_h_refs,autolink_build_artifacts,rewrite_build_artifact_paths), so flat→nested is a full link-layer reimplementation (verify by new-broken⊆baseline), not a config migration. 4 bespoke builders to port as hooks (copy_build_artifactscopies PDF+PNG+tables tofigure/+table/,autolink_build_artifacts, ~300-linebuild_key_artifacts_page, note asset-folder copies) — the artifact copy needs a pre-docs hook (copy_site_figuresis pre-docs butconfig.hooksare post-docs); addSiteConfig.pre_docs_hookswith ficha as its first consumer (don't land it unexercised). Verifiable-in-sandbox parts: docs, 82 analyses, hypotheses, findings, subdirs, 231 script pages, 2 dataset pages, key_artifacts. Fold in a check of the possibly-staledocs/reference/analysis-index.yaml.saude— empirical; ~30% maps to stock (top-level docs + stock cached paper). LIVE bespoke subsystems needing hooks: a news timeline (references/news/stories.csv→news.html) and an 18-source references index (references/→references.html); notebooklm is dead (droppable). Dataset portal reads an empty summary cache in-sandbox → needs a data host to regenerate the cache and verify dataset pages.rolis intentionally NOT migrated (decision 2026-08-31, Henrik): it's not a docs site but a bespoke claim-graph knowledge base (14 typed node types, strand cross-cut indexes, an interactive graph explorer + generatedsite-data.js, a linear book). Its deploy is already on the sharedsite_deploy.sh, and a graph archetype would be single-consumer code in shared sitekit, so rol stays on its ownbuild_all.py. Do not force it ontominimal(catastrophic content loss) and do not re-open this without a second graph consumer to justify the archetype.
For unmigrated projects the old archetype-reference workflow is still
authoritative: copy build_all.py + templates/ from the matching
canonical reference and customize. For new projects, prefer the sitekit
path (see "Using sitekit" below).
The empirical archetype (fisc, poll-sponsor-bias, connect's docs-heavy
variant) and the theoretical archetype (bind — cases discovery +
per-case pages via a project cases.html, grouped cases index,
extra_tex_pages for holdings-style companion LaTeX, plus a citations hook)
are both real and exercised, as are the AN-page / cite-ref / anecdote /
findings / hypotheses / script-page passes. Still a stub: any graph
archetype (rol). sitekit also gained SiteConfig.exclude_stems (keep
specific subdir notes off a PUBLIC site) during this work.
Arguments
/site-- create site scaffold + build for the current project/site [project-slug]-- target a specific project
What the site contains
- Index page -- landing page with paper/talk hero cards and docs grouped by category
- Doc pages -- each
docs/*.mdfile rendered as HTML with a table of contents linking to h2/h3 sections, image lightbox, anchor linking, Hypothes.is annotation layer - Paper page -- LaTeX paper converted to HTML via make4ht, with inline footnote tooltips, MathJax, Hypothes.is annotation layer
- Talk page -- beamer slides converted to HTML via make4ht
Step-by-step
1. Locate project
Find workspace root by searching upward for CLAUDE.md next to projects/ and pipelines/.
Resolve the target: $ROOT/projects/{slug}/ or $ROOT/pipelines/{slug}/ — a slug can
name either. If /site foo is given, check both trees; pipelines are as valid a target as
projects.
Read the target's CLAUDE.md to get the title and short description.
Pipeline repos differ from project repos in three ways — mind them throughout:
- Deploy plumbing. Pipeline
build.shsources the sharedresearch-kit/tools/site_deploy.sh(providingsk_encrypt_site/sk_deploy_site) and setsSITE_TITLE+ aSITE_GATEDtoggle, rather than inlining the staticrypt/rsync logic. Public data-reconstruction pipelines setSITE_GATED=0(plaintext, e.g.lovhistorie); pipelines documenting access to sensitive/geoblocked raw data setSITE_GATED=1(default — gated, e.g.govspend). Model a new pipelinebuild.shonpipelines/lovhistorie/build.sh(public) orpipelines/govspend/build.sh(gated), NOT onserasa/build.sh. source/may not be a package. Pipelines often run scrape/clean scripts directly, sosource/__init__.pycan be absent;python3 -m source.site.build_allneeds it. Create emptysource/__init__.pyandsource/site/__init__.pyif missing.- Usual archetype is
minimal(docs-only) orempirical(data portal) — a pipeline rarely has apaper/, so setpaper_title=""and apaper_placeholder_msgthat says "this is a pipeline".lovhistorieis the canonical minimal pipeline;govspendthe canonical empirical pipeline (source cards over raw + clean tables — see "Empirical data portal for a pipeline" below).
2. Check if site already exists
If source/site/build_all.py already exists:
- Ask the user if they want to rebuild only (run the existing script) or regenerate the scaffold.
- If rebuild only: skip to step 5.
3. Create site scaffold
Create the following files using the canonical templates below.
Directory structure
source/site/
build_all.py # main generator script
templates/
index.html # landing page
doc.html # docs page (markdown rendered)
paper.html # paper page (make4ht content)
talk.html # talk/slides page (make4ht content)
Customization points
When creating the scaffold, customize these project-specific values:
PROJECT_TITLE-- short title for the nav brand (e.g., "Causal Judge", "Corruption Networks"). Read from project'sCLAUDE.mdfirst line heading ordocs/summary.md.PAPER_TITLE-- full paper title for paper/talk page headers. Read frompaper/main.texorpaper/paper.tex(\title{...}), or fall back todocs/summary.mdheading.DOC_REGISTRY-- list of (path, title, description, category) tuples. Scandocs/for existing.mdfiles and register them using the standard mapping:
| File | Title | Category |
|---|---|---|
| summary.md | Research Summary | Reference |
| institutions.md | Institutional Background | Reference |
| data.md | Data Sources | Reference |
| methods.md | Methods | Reference |
| literature.md | Literature | Reference |
| thinking.md | Open Questions & Ideas | Working notes |
| decisions.md | Key Decisions | Working notes |
| outline.md | Paper Outline | Working notes |
| hypotheses.md | Hypotheses | Working notes |
| desiderata.md | Desiderata | Working notes |
| todo.md | Active Tasks | Tasks |
| done.md | Completed Tasks | Tasks |
| meetings.md | Meeting Notes | Communication |
| feedback.md | External Feedback | Communication |
Only include files that actually exist in the project's docs/ directory.
If there are .md files not in this table, add them with a sensible title and the "Reference" category.
4. Write the files
Using sitekit (preferred path for new projects and serasa-style minimal projects)
Two files in source/site/:
# source/site/build_all.py — thin shim
import sys
from pathlib import Path
_SITEKIT = (
Path(__file__).resolve().parent.parent.parent.parent.parent
/ "research-kit" / "sitekit"
)
if str(_SITEKIT) not in sys.path:
sys.path.insert(0, str(_SITEKIT))
from sitekit import build_site
from source.site.site import config
if __name__ == "__main__":
raise SystemExit(build_site(config))
# source/site/site.py — config + project-local hooks
from pathlib import Path
from sitekit import SiteConfig
config = SiteConfig(
project_root=Path(__file__).parent.parent.parent,
project_title="…",
paper_title="…",
archetype="minimal", # or "empirical" / "theoretical" / "mixed"
paper_tex="paper.tex",
talk_tex="talk.tex",
doc_registry=[
("docs/summary.md", "Research Summary", "...", "Reference"),
# ...
],
)
The sitekit package bundles default templates/. A project can override any
one by placing a same-named file under its source/site/templates/: project
files take precedence over package files.
When the project needs an extra section that doesn't generalize (e.g.
serasa's specs-for-Ramon dropdown, BCB results gallery), write it as a hook
callable in site.py and add it to config.hooks. The hook receives a
BuildContext and may return {"extra_cards": "<html>"} to inject content
above the index's standard doc-group grid.
On educloud the system venv is read-only, so the sys.path shim in
build_all.py replaces pip install -e. On a writable env you can
substitute pip install -e ../../research-kit/sitekit and drop the shim.
Empirical data portal (archetype="empirical") — extra wiring
The data portal (source-card landing grid → per-source pages → per-table pages
with columns / example rows / histograms) needs more than the site.py shim.
Gotchas, learned building govspend (canonical empirical pipeline):
- sitekit does NOT bundle
dataset.html,source.html, orindex.html. The empirical archetype raisesTemplateNotFoundError: dataset.htmlunless the project supplies them undersource/site/templates/. Copy all three fromprojects/fisc/source/site/templates/(the canonical empirical set), then edit onlyindex.html's<title>andsummary-boxfor the new project. The bundledindex.htmlhas NO<!-- INJECT_SOURCE_CARDS -->marker, so without a copiedindex.htmlthe landing page silently omits the source cards even though the source/dataset pages build fine — always verify the index actually linkssources/*.htmlafter building. - Two-stage build.
source/summary/{config,compute,build_all}.pycomputes a per-dataset JSON cache (source/summary/cache/*.json, committed to git so the site rebuilds without raw data); the site reads that cache.build.shneeds asummarymode (python3 -m source.summary.build_all [--force|--only <id>]) separate fromsite. Modelconfig.py/compute.pyonprojects/fisc/source/summary/. config.pycontract:SOURCES(list ofSourceConfig(id, name, description, categories)) +DATASETS(list ofDatasetConfig, each with acategoryin exactly one source'scategories) + aCACHE_DIR. Source group pages only build if every source has non-emptycategories.- Big / heterogeneous data (pipeline-scale): fisc's
compute.pydoes a singleread_parquet— fine for projects, not for a pipeline with 500M-row partitioned dirs or JSONL raw.govspend'scompute.pyis the reference for: exactrow_countfrom Parquet footer metadata (no data read) + a stratified sample of parts spread across the partition list (NOT the first N — a{uf}-{year}layout sampled first-N yields an all-Amazonas histogram); skip 0-row parts (BigQuery exports leave padding files that zero out stats); a JSONL reader; and CPF masking ([CPF]for CPF-shaped values + amask_columnslist for person-name columns) — do this even on a gated site. Firm CNPJs/names stay visible; individual CPFs never surface. Note CEIS/CNEP-style public naming registries are the exception — leave party names visible, mask only CPFs. Record astats_scopenote (which/how many parts scanned). - Full-text corpora (gazettes, court text, etc.) OOM a naive read. A
textcolumn of multi-KB documents times the scanned rows is gigabytes — a whole-partread_parquetgets the process silently OOM-killed (looks like an empty exit when piped throughgrep; symptom: only the first 1–2 caches written, no error).diario-municipal'scompute.pyis the reference: read a row-bounded first batch viapq.ParquetFile(f).iter_batches(batch_size=…)(a20k-row budget spread across the stratified scan parts) instead of loading whole parts, and truncate sample-row cells (240 chars) so a dataset page doesn't embed megabytes of raw act text. When a build "completes" with a near-empty cache and no error, suspect OOM on atextcolumn — run the build in the foreground (unpiped) to see the real output. - Raw vs clean pages: show a raw table page only where the clean layer drops
fields the raw carries (or the raw has no clean equivalent); otherwise the clean
table is the faithful view. Channels whose raw is ZIP/nested dumps (not cleanly
tabular) are represented by their clean tables with a feeder note in
source_notes. - Turn off
build_descriptives/build_tablesinSiteConfigfor a pipeline with no paper-stylebuild/figure+build/table; the data portal is the content. A stray aggregatebuild/figure/*.pngstill gets copied intobuild/site/figures/(unencrypted on a gated site) — confirm it carries no PII or drop it. - Coverage checkerboard (high-value landing hero). Both
govspendanddiario-municipalship asource/summary/coverage.py(writes a committedcoverage.json: a UF × data-type × year cube) +source/figure/ coverage_matrix.py(small-multiples — each cell a mini year-histogram on a fixed shared x-axis, bar height ∝ √count normalized within column, tri-state green/amber/blank color), embedded on the index via an<img>the copiedindex.htmltemplate references. Copy both. Compute the cube cheaply/exactly: UF and year from{uf}-{year}partition filenames + footers where possible, else read just the uf + year/date columns; derive UF fromibge7(first two digits = IBGE state code) when there's no uf column. Pick the amber="partial" semantics per pipeline: govspend uses it for platform-limited/incomplete columns (BLL, aditivos), diario for recall-limited derived extractions vs complete corpus text. Name capability columns honestly ("Bids · incl. losers", not "full rolls") and verify surprising ones against the data before shipping.
Using the legacy fork-and-customize pattern (for archetype-rich projects until they migrate)
Pick the canonical reference that matches the project's archetype, then copy
its build_all.py and templates/ as a starting point. Both archetypes share
the same design system (palette, typography, nav, summary-box, Reading Guide,
All Documentation) — they differ only in what middle sections appear on the
landing page and what auxiliary templates exist.
| Archetype | Canonical reference | What's distinctive |
|---|---|---|
| Theoretical | /home/henrik/research/projects/bind/source/site/ |
Cases section + per-case pages; Results at a Glance propositions table; briefs |
| Empirical | /home/henrik/research/projects/fisc/source/site/ |
Data Sources grid + per-source / per-dataset pages; Descriptives + Tables pages |
How to tell which archetype:
- Theoretical if the project's
paper/is a formal model (propositions, theorems, proofs), the docs includepropositions.md/holdings.md/ case extractions, and there's nodata_catalog/-backed parquet dataset list. - Empirical if the project has descriptive figures/tables backed by data
pipelines,
source/figure/andsource/table/scripts, and asummary/cache of dataset metadata. - Mixed projects exist (e.g. an empirical paper with a small theory appendix). Pick the archetype that matches the primary output and add the secondary archetype's sections on top of it.
Then customize what you copied:
build_all.py: updatePROJECT_TITLE,PAPER_TITLE,DOC_REGISTRY, and the.texfilenames if they differ. Drop archetype-specific scaffolding the new project doesn't need (e.g.AREA_MAP/TOPIC_MAP/cases.htmlif you copied bind into an empirical project). Also make sure it: wipesSITE_DIRat the start of each build (so renamed/removed sources don't linger as stale files); does not copypaper.pdf/talk.pdfinto the site (staticrypt encrypts only HTML — a shipped PDF would be readable by direct URL; the make4ht HTML render is the paper page); and contains no deployment logic of its own (noDEPLOY_DIR, no rsync to~/hsigstad.github.io— deployment isbuild.sh's job, see step 6).- Cite-ref machinery: if the project has (or will have) a
docs/literature.mdordocs/literature/index.md, the build must turn[cite:<bibkey>]tokens into hyperlinks perresearch/rules/citations.md. The canonical implementation lives in/home/henrik/research/projects/connect/source/site/build_all.py— copy these helpers verbatim:_load_cite_map()+CITE_MAP(papers with their owndocs/literature/<key>.mdpage)_load_bib_authoryear()+BIB_AUTHORYEAR(author-year labels parsed frompaper/*.bib)_load_index_cite_map()+INDEX_CITE_MAP(papers that appear only as[cite:<key>]-tagged bullets on the literature index)_link_cite_refs(html, current_stem)— resolution order: page → index anchor → literal token_inject_index_cite_anchors(html)— addsid="cite-<key>"to<li>elements on the literature/index page Then wire_link_cite_refsinto the render pipeline (alongside the existing_link_an_refs/_link_h_refs/_link_anec_refspasses) and run_inject_index_cite_anchorsonly when renderingdocs/literature/index.md. The CSS rule.md-body li:target { background: #fff8c4; scroll-margin-top: 4rem; }intemplates/doc.htmlhighlights the bullet you jumped to.
- Templates: update
<title>tags and any paper/talk<h1>to use the new project's title. Keep all CSS in:rootand surface colors as-is — they are part of the contract below.
The canonical projects are the single source of truth for the templates; do not inline them in this skill. If you find a fully built site that diverges from the cream palette / nav style / summary-box+Reading Guide skeleton, that's drift — bring it back in line with the design-system invariants below.
Design system invariants
Every site must preserve these — they are what makes the projects look like one cohesive workspace. If you copy from any project that lacks them, fix it to match.
Color tokens (in every template's :root):
:root {
--bg: #faf9f6; /* warm cream background */
--card: #fffefa; /* slightly lighter cream for cards/dropdowns/page-headers */
--fg: #374151; /* soft near-black text */
--muted: #6b7280; /* secondary text */
--border: #e5e2db; /* warm cream border */
--accent: #2563eb; /* blue links / left-borders / primary actions */
--accent2: #198754; /* secondary green accent (rarely used) */
}
Surface colors that pair with the cream palette (use these instead of
cool-gray #f8f9fa / #f0f4f8 / #eef1f5 / #fafbfc):
- Code-inline background:
#f0ede6 - Code-block / blockquote / table-header background:
#f5f3ee - Even-row striping:
var(--bg)(#faf9f6) - Hover surfaces (table rows, dropdown items):
#f5f3ee(light) or#f0ede6(slightly stronger)
Body text uses Georgia/Times serif at 1.06rem / line-height 1.78 on
.md-body / paper / brief / case pages; UI chrome (nav, headings, page-header,
table headers) stays on the system sans-serif.
Sticky top nav is background: #212529 with white brand on the left and
muted (#adb5bd) link text that brightens on hover. Active link is full white.
Landing page (index.html) structure — every project's landing page
follows this top-to-bottom shape:
<div class="summary-box">— one paragraph executive summary, with a blue 4px left border. Lives just inside<main>.<h2>Reading Guide</h2>followed by<div class="guide-grid">— a 2-column grid ofguide-cardlinks, each with a small uppercase priority label above the title (priority-startred,priority-mainblue,priority-refgray) and a one-line description below. The first 1–2 cards are the "start here" entry points (paper, talk, lead brief).- Archetype-specific sections — see below. Reuse
summary-box/ table / card vocabulary; never invent a new card style per project. <h2>All Documentation</h2>followed by<div class="doc-groups">— doc cards grouped by category, each group with a colored 3px top border.
Page width is max-width: 900px for the index, 42rem for doc/brief/case
pages. The body font is .88rem on the index, 1.06rem (Georgia serif) on
content pages.
Archetype-specific sections
The middle of the landing page (step 3 above) and the set of auxiliary
templates differ by archetype. Match these to whatever copy of build_all.py
you started from.
Theoretical projects (e.g. bind) typically include:
- Results at a Glance — one or two
<table class="results-table">blocks listing propositions/corollaries with a one-line "what it says" column and a<span class="badge badge-proved">/badge-draftstatus pill. Source: bindindex.htmllines 142–173. - Decisions Needed —
<ol class="decision-list">with each<li>styled as an amber-bordered card auto-numberedD1, D2, …via CSS counters. - Cases (or analogous content section): a separate
cases/index.htmlpage grouped by area/topic, plus per-case pages built fromcases/extractions/*.md. Add a "Cases" dropdown to the nav. - Auxiliary templates:
cases.html(grouped grid), and a paper template that handles multiple.texdocuments (paper + holdings, etc.).
Empirical projects (e.g. fisc) typically include:
Data Sources —
<div class="data-grid">ofdata-cardlinks, one per source group, with adata-metarow-count badge ('533K','9.8M'). Each card links tosources/{id}.html.Descriptives / Tables doc pages — auto-generated from
build/figure/PDFs (converted to PNG) andbuild/table/*.mdfiles; the build script appends them toDOC_REGISTRYunder a "Descriptives" category.Auxiliary templates:
dataset.html(Chart.js histograms + column-info table + sample-rows table),source.html(group overview with category rollups), and optional per-source narrative templates undertemplates/sources/{id}.htmlfor groups that have no parquet backing (e.g. a docs-only "diários" page). Thedataset.htmltemplate includes a Binary columns card above the categorical block: one horizontal bar per column showing the share of "true" (1 / "S" / "Y"), with NA% annotated per row. Binary columns are auto-detected bysource/summary/compute.py::detect_binaryfrom bool dtype, integer ⊆ {0,1,NA}, and string ⊆ {"S","N"} / {"Y","N"} / etc. — no manual registration. Detected binaries are excluded from the per-column value-count chart to avoid double-display.Canonical source of the binary helper: snippets live in
snippets/next to this skill —snippets/detect_binary.pyfor the Python detection logic andsnippets/binary_chart_block.htmlfor the JS chart block. When scaffolding a new project (or refreshing one whosecompute.py/dataset.htmlpredate the binary feature), paste from these files rather than copying from a random project. When updating the helper, edit the snippet first and propagate to projects.Example-value helper: the column-info table on each dataset page has an "Example" column showing one sampled non-null value per column. Canonical helper:
snippets/example_value.py. Adds anexamplefield to each entry in thecolumnslist of the cache JSON; the template renders it as the rightmost cell in the column-info table.Source-script pages: every
.py/.R/.sh/.sqlundersource/is rendered to its own HTML page atbuild/site/source/<subpath>.htmlwith per-line anchors (#L42). The dataset page's "Source:" line links to it (project-owned scripts only — upstreampipelines/*references stay plain text). Canonical helper:snippets/source_pages.py(functions forfind_source_scripts,script_out_name,build_source_pages, plus the per-line gutter CSS fordoc.html).Pseudocode block: assembled datasets get an optional
DatasetConfig.pseudocodefield — a short, hand-written summary of how the dataset is built, with input-dataset references in[brackets]that the template rewrites to hyperlinks. Renders as a<pre>block near the top of the dataset page. Canonical helper:snippets/pseudocode_block.md. Lets the long description / notes paragraphs come down — the construction logic lives in one place where every input is clickable.Grain-verification helper: when a
DatasetConfig.key_columnsfield is declared (e.g.key_columns=["cpf", "npu"]), the dataset page renders a tri-state badge in the page header — green ✓ "Unique on (...)" when the key holds, yellow "!" "Unique but null keys on (...)" when there are no duplicates but some rows have null values in a key column (typical for left-joined panels), red ✗ "Grain broken on (...)" when duplicate rows exist. Canonical pair:snippets/verify_grain.py(Python helper that emits agrain_checkblock in the cache JSON) andsnippets/grain_check_badge.html(the page-header JS block). The point is end-to-end data-quality observable on the page itself: if a future change to the assemble script silently breaks grain, the badge flips on the next cache rebuild — no separate ledger entry to maintain.Talk page (
talk.html) — empirical projects usually have a beamer talk alongside the paper; theoretical ones often don't.
Mixed: layer the secondary archetype's sections on top of the primary. E.g. an empirical paper with theory: start from fisc, add a Results at a Glance table for the theoretical results above the Data Sources section.
5. Build the site
Run:
cd $PROJECT_ROOT && python3 -m source.site.build_all
Report what was generated (number of doc pages, whether paper/talk were built).
6. Deploy: staticrypt-encrypted, to the project's gh-pages branch
This is the canonical deployment for every project: the site is
staticrypt-encrypted and pushed to the project's own gh-pages branch. One
self-contained command, a readable repo-name URL
(https://hsigstad.github.io/{repo}/), and a password gate so the private
research content isn't readable by anyone who stumbles on the URL. There is no
separate plaintext push or ~/hsigstad.github.io/{slug}/ personal-website
step — that legacy approach is retired.
The reference implementation is /home/henrik/research/projects/serasa/build.sh
(self-contained, project style). Port it (adjusting PROJECT_TITLE and the
paper/talk .tex names) to any project that doesn't have it yet — including
projects still on the old plaintext gh-pages push or the legacy
~/hsigstad.github.io/{slug}/ rsync.
Pipeline repos take the shared-helper route instead (see step 1): their
build.sh sources research-kit/tools/site_deploy.sh and calls its
sk_encrypt_site / sk_deploy_site — no inlined staticrypt/rsync. Set
SITE_GATED=0 for a public pipeline (pipelines/lovhistorie/build.sh) or leave
it gated (default 1) for a sensitive one (pipelines/govspend/build.sh). Same
gh-pages / repo-name-URL / password-gate outcome as below.
The self-contained build.sh wires three functions into a deploy mode:
build_site— make4ht for paper/talk, thenpython3 -m source.site.build_all→build/site/(plaintext).encrypt_site— staticrypt overbuild/site/→build/site-encrypted/:
staticrypt keeps the input dir's basename, so flattenSTATICRYPT_PASSWORD="$pw" npx --yes staticrypt build/site \ --recursive --directory build/site-encrypted \ --config build/.staticrypt.json \ --short --template-title "<PROJECT_TITLE>" \ --template-instructions "Enter the shared password to access the site." \ --remember 30build/site-encrypted/site/up one level afterward.--config build/.staticrypt.jsonkeeps the salt artifact out of the repo root.deploy_site— clone thegh-pagesbranch to a tmpdir (orphan-init it if absent),rsync -a --deletefrombuild/site-encrypted/(notbuild/site/), writerobots.txtwithDisallow: /, commit, push.
deploy mode chains them: build_site; encrypt_site; deploy_site. Run with:
cd $PROJECT_ROOT && bash build.sh deploy
Requirements: npx (for staticrypt) and git push access to the project repo.
site_deploy.sh clones/pushes gh-pages over SSH (git@github.com:…)
regardless of the repo's origin URL, so the deploy works even when origin is
HTTPS. But committing the source to an HTTPS-origin repo fails here (no
credential helper: could not read Username for 'https://github.com') — push via
token instead: git push "https://x-access-token:${GH_TOKEN}@github.com/<owner>/<repo>.git" HEAD:main
(leaves origin untouched). SSH-origin repos push normally.
The site password. Each project gets its own, stored in a gitignored
.site-password at the project root (also read from $STATICRYPT_PASSWORD).
Pick a memorable phrase tied to the project's topic, in three-word
hyphen-slug form (aaa-bbb-ccc) — for Brazil law/economics projects, a
Portuguese legal term or a phrase from the subject matter. Example: serasa uses
dano-moral-presumido (the "presumed moral damage" doctrine central to its
cases). When setting up a new site, propose a password in this style, write it
to .site-password, and tell the user what it is — it is not secret from the
user, they need it to view the site. Add both .site-password and
.staticrypt.json to the project .gitignore.
7. Verify
Check that build/site/index.html exists and list the generated files. After a
deploy, confirm https://hsigstad.github.io/{repo}/ serves the staticrypt
password gate (look for class="staticrypt-html" / "Enter the shared password")
— on a sub-page too, not just the index.
Important rules
- Never modify templates in other projects -- each project gets its own copy.
- The design system (CSS, nav bar, JS) must be identical across all projects for visual consistency.
- All sites include
robots.txtwithDisallow: /and<meta name="robots" content="noindex, nofollow">-- these are private research sites. - The deployed site is staticrypt-encrypted and pushed to the project's
gh-pagesbranch -- see step 6.bash build.sh deployis the one canonical deploy path; there is no plaintext or personal-website publish step. - Never ship sensitive non-HTML assets into
build/site/-- staticrypt encrypts only HTML, so PDFs/CSVs/images would be readable by direct URL. The only non-HTML file in a deployed site should berobots.txt. - Paper/talk pages are optional -- if
build/make4ht/doesn't exist, the build skips them gracefully and shows a placeholder card on the index. - Don't add data portal features (dataset cards, Chart.js) unless the user explicitly asks. The default is the docs-only pattern.