Notcrawl

Notcrawl

Use local archive data first for Notion questions. Browse or hit the Notion API only when the archive is stale, missing the requested scope, or the user asks for current external context.

Sources

• DB: ~/.notcrawl/notcrawl.db
• Config: ~/.notcrawl/config.toml
• Cache: ~/.notcrawl/cache
• Markdown archive: ~/.notcrawl/pages
• Git share repo: ~/.notcrawl/share
• Repo: ~/GIT/_Perso/notcrawl
• Preferred CLI: notcrawl; fallback to go run ./cmd/notcrawl from the repo if the installed binary is stale

Freshness

For recent/current questions, check freshness before analysis:

sqlite3 ~/.notcrawl/notcrawl.db \
  "select coalesce(max(synced_at), 0) from sync_state;"

Routine refresh:

notcrawl doctor
notcrawl sync --source desktop

API refresh:

notcrawl sync --source api

Use notcrawl sync --source all only when both desktop and API sources are configured and the broader refresh is intentional.

Query Workflow

1. Resolve scope: workspace, teamspace, page, database, author, keyword, or date range.
2. Check freshness for recent/current requests.
3. Use CLI for normal reads; use read-only SQL for precise counts/rankings.
4. Report absolute date spans, counts, page/database titles, and known gaps.

Common commands:

notcrawl search "query"
notcrawl databases
notcrawl report
notcrawl sql "select count(*) from pages;"

SQL

Use notcrawl sql for exact counts, joins, and database/page inventory queries when normal CLI reads are too coarse. The command only allows read-only select, with, and pragma queries.

Useful examples:

notcrawl sql "select count(*) as pages from pages;"
notcrawl sql "select parent_table, count(*) as pages from pages group by parent_table order by pages desc;"
notcrawl sql "select title, last_edited_time from pages order by coalesce(last_edited_time, created_time, 0) desc limit 20;"

Do not use SQL to mutate the archive.

When the installed CLI lacks a new feature, build or run from ~/GIT/_Perso/notcrawl before concluding the feature is missing.

Notion Boundaries

Desktop mode snapshots the local Notion SQLite database read-only and must not write to Notion application storage. API mode requires NOTION_TOKEN; do not invent token availability. Git-share snapshots and Markdown exports must not include secrets.

Verification

For repo edits, prefer existing Go gates:

GOWORK=off go test ./...

Then run targeted CLI smoke for the touched surface, for example:

notcrawl doctor
notcrawl status --json
notcrawl search "test"