Scry Skill
Scry is read-only SQL (ClickHouse dialect) over registered public corpora
— Hacker News, Reddit, the Twitter archive, books, papers, forums, SEC
filings, the crawl — one call from a question to cited rows. Queries are
free while the system has slack: every response reports billing_mode
and spend_nanodollars, and the money arguments (x-scry-budget,
x-scry-max-seconds; MCP budget_nanodollars, max_seconds) are
ceilings you choose, never fees. Ask your wildest curiosity.
Three one-call questions (POST /v1/scry/query with Content-Type: text/plain, or the MCP sql tool). The first Hacker News item to
mention bitcoin:
SELECT hn_id, original_author, original_timestamp, title
FROM hackernews.items
WHERE hasToken(search_text_lc, 'bitcoin')
ORDER BY original_timestamp ASC
LIMIT 5
Who said "vibe coding" before Karpathy:
SELECT tweet_id, original_timestamp, text
FROM twitter.tweets
WHERE hasAllTokens(search_text_lc, ['vibe', 'coding'])
AND positionCaseInsensitive(search_text_lc, 'vibe coding') > 0
AND original_timestamp < '2025-02-01'
ORDER BY original_timestamp ASC
LIMIT 5
Where Reddit talked bitcoin in 2013:
SELECT subreddit, count() AS n
FROM reddit.comments_popular
WHERE created_utc >= '2013-01-01' AND created_utc < '2014-01-01'
AND hasToken(search_text_lc, 'bitcoin')
GROUP BY subreddit
ORDER BY n DESC
LIMIT 10
Every response carries rows, read_rows, coverage,
deadline_partial, truncated, and the meter (burden_nanodollars is
what the machine did, spend_nanodollars what you paid). A cut scan
(deadline_partial: true, or a deadline error) wants a rarer token, a
tighter WHERE or LIMIT, or a smaller sibling relation
(reddit.comments_popular beside reddit.comments, x_open.tweets
beside twitter.tweets); the x-scry-explain: 1 header (MCP explain: true) pre-flights a wide statement for free — the index analysis returns
and nothing runs.
Search like the answer exists. It almost always does — under a
vocabulary, a venue, or an era you have not probed yet — so treat every
empty result as a wrong probe before treating it as an absence. You are
covering a space, not fetching an answer: fan vocabularies, sweep
relations, cross time windows, run lexical and semantic arms in
parallel, chase edges, and keep going past the first sufficient-looking
hit — the tenth probe is where a field opens. Done is saturation — new
probes returning only rows already seen — never satisfaction. Report
the space covered, not just the hits.
The live schema is the contract; static relation lists are only
orientation.
Skill generation: 2026082203
Workflow
- Load the durable key from
~/.config/scry/env (legacy ~/.scry/.env
still honored). Context is readable without a
credential; schema, stats, and queries require your key. When the Scry
MCP server is connected (the ExoPriors/skills plugin wires
mcp.scry.io on install), use its tools directly — the OAuth
connection is the credential and no key file is needed; the key path
below serves raw HTTP. If neither an MCP connection nor a key is
available, stop before going further and direct the user to
https://scry.io/#console.
- Call
GET /v1/scry/context?mode=agent&skill_generation=2026082203.
For worked, measured query shapes, GET /v1/scry/examples?mode=index
(free, no key) lists the query-complexity tree one row per entry —
every entry introduces exactly one construct atop its parent's, from
selectivity probe to semantic ANN, each with its observed wall time and
the byte size of its SQL. ?slug=<slug> fetches one entry's problem,
SQL, technique, and measurement; ?mode=tree nests the taxonomy,
?mode=chains lists root-to-leaf ladder walks; the bare route returns
every entry in full (144 KB).
- Discover from the doors. The default
GET /v1/scry/schema document
already carries full contracts for the primary-tier doors plus a compact
depth_relations index of every supporting table; fetch further full
contracts with GET /v1/scry/schema?relation=<name>[,<name>], or
?mode=index for the whole catalog as one relation | tier | extent | lag | purpose line per relation (both also exposed as the MCP schema
tool's mode and relation arguments; the MCP default is the index and
mode="contract" carries the product contract, census, and live
statistics). Use only
relations and helper functions returned there, and read each relation's
query_guidance block — filter_columns_first, indexed_predicates,
coverage_note — before writing the first predicate: it names the
indexed access paths. Never guess column names from memory of similar
sources — a wrong column returns the relation's real column roster in
the error, so one failed query self-corrects in one step; an unknown
relation returns the nearest registered names.
- Send one SQL statement to
POST /v1/scry/query with
Content-Type: text/plain.
- Semantic search: mint a named query vector with
POST /v1/scry/embed
{text, name}, then use it as the unquoted @name inside
scry_vector_topk_distance; full patterns are in
references.md § Scry query patterns. Query text craft dominates
every other parameter: embed answer-shaped, exuberant passages —
the paragraph you hope to find — never keyword stubs, and fan out
registers (references.md § Writing the query text). The same endpoint takes
{expression, name} to compose stored handles (contrast axes,
centroids, debiasing) into a new saved handle with diagnostics —
see references.md § Composing embeddings into saved handles and
the schema's vector_recipes. The ANN set is dynamic — a relation
leaves it while its vector index re-materializes — and the schema names
the live set: only surfaces with serves_ann: true accept ANN ranking
(the rest still serve plain SQL). ANN queries must be standalone (no
JOIN); hydrate companion text in a second query.
- Keep every query bounded with
LIMIT. Start at 20 and widen only after
inspecting row shape, provenance, and source coverage.
Token search speed is governed by the rarest token: in
hasToken/hasAllTokens filters include at least one distinctive
token (a name, identifier, or unusual word) — all-common-word token
sets scan a large share of the table and run 30-60s. A slow query's
response carries a performance_note naming the fix. For broad
topical questions with only common words, use the embeddings helpers
instead.
- Parse results from
rows, not a data key: each row is a plain JSON
array with values in column order. A client that reads data sees
false empty results.
Memory
Scry hosts one cross-platform memory document per account
(MCP memory/memory_write):
markdown, default slug main, 64KB, shared by every agent and harness the
user connects. At session start read it alongside context (version 0 +
empty content = none yet). At session end, consolidate durable user
preferences — including what worked against Scry: relations, query
patterns, vector handles — back into it under a ## Scry usage heading.
Writes are whole-document compare-and-swap on if_version; a 409 returns
the current head — merge into it and retry. Keep it compressed: the cap is
the decay function. If the document is empty and the user's local agent
memory holds durable preferences, you may offer — once, and only with the
user's explicit approval — to consolidate them into Scry memory so they
travel across platforms. Encrypted at rest server-side.
Do not use engine catalogs, foreign-dialect casts or operators, compatibility
helpers, or a fallback corpus database. Do not invent relations. Pass a
search-grammar line as q to MCP sql; SQL remains the only read verb.
The q search grammar speaks a full lexical language: bare words AND
together; "exact phrase"; a OR b; -term / -"phrase" exclusion;
( ) grouping; /pattern/ regex over full text (case-insensitive,
negatable; RE2 only — SQL rejects lookaround and backreferences rather than
counting a prefilter's superset. A positive literal or token anchors the
query; rust /[0-9]+/ can use rust to bound the regex residual, while
bare /[0-9]+/ is refused); word* wildcards; word~1 fuzzy
(typo-tolerant: a 4-24 char word resolves against the corpus vocabulary
into its real one-edit word forms and searches as their OR —
query_plan.clamped echoes the forms chosen; bare ~ means ~1,
larger asks clamp to 1 with a note); "exact phrase"~3 slop
(phrase words in order, at most N intervening words between neighbors,
max 50); and
a NEAR b / a NEAR/50 b proximity (uppercase NEAR; matches both orders
within N characters, default 100, max 1000; operands may be words, quoted
phrases, /regex/, or (x OR y) groups). Substrings and CJK phrases can
use a sufficiently built n-gram index; read the relation's capabilities,
not a corpus-wide availability claim.
MCP sql with q requires one registered relation, never "*".
It returns ordinary SQL rows and the executed compiled_sql; it does not
silently weaken a zero-result query. Inspect that SQL before interpreting
membership. With explain: true, the statement is validated and its
ClickHouse index analysis is returned without executing the corpus query,
beside a forecast — rows_est, bytes_est_uncompressed and seconds_est
from the measured rows and bytes per granule and the measured scan rate,
fits_max_seconds against the deadline the call would run under, and
faster (sibling relation plus the rewritten statement) when it does not.
Request prompts/get with name: "query_guide" and tool: "sql" for
composition patterns and the current input schema.
The compiler's internal plan distinguishes declared indexes from measured
coverage: zero-built word indexes do not establish pruning, and partial
coverage is not complete coverage. EXPLAIN is the actual plan evidence,
especially for views whose backing indexes are not mapped in discovery.
Use bounded, independently recorded queries to compare several relations;
the MCP SQL tool does not accept a multi-relation grammar sweep.
The grammar is also a first-class SQL operand: inside any
POST /v1/scry/query statement, scry_lex('<line>') expands
server-side into exactly the predicate sql with explain would return for
the statement's one registered relation — so
WHERE scry_lex('"scaling laws" -toy'),
countIf(scry_lex('/GPT-[0-9]/')) AS hits, and GROUP-BY histograms
over a lexical cohort are plain SQL. An optional second argument pins
the text expression (scry_lex('rust', title)); an operator the
relation cannot express is a hard error, never a silent drop. At most 8
calls per statement; one registered relation per statement.
Lexical recipes
Reuse shared term instruments with scry_recipe('<slug>'[, text]) for
membership and scry_recipe_score('<slug>'[, text]) for token-weighted
score. Use scry_recipe_density('<slug>'[, text]) for weighted term
occurrences per 1,000 characters across token, phrase, and regex members.
Discover them with MCP recipes; publish a complete measured
version with recipe_write and the returned head version as
if_version. Derive candidates read-only with recipe_derive, then curate noise, measure the instrument, and publish through recipe_write. Write a recipe when you derived at least five surface forms,
or when a polarity instrument survives reading 20 matches per cohort.
Read those matches before publishing, keep provenance and measurements
with the terms, and treat the stance as part of the recipe's identity.
The seeded shelf and choosing guidance live in references.md § The
recipe shelf; the author/thread/time/graph quantifier shapes that
recipes plug into are references.md § The quantifier chain; the full
plane-by-plane operator map — quorum and frequency gates, named
quantifiers, Allen span relations, life-history regex, epistemic
operator families — is references.md § The operator space.
Composing recipes has an operand: scry_recipe('a - b') difference,
scry_recipe('a & b') intersection, scry_recipe('a ^ b')
exclusive-or — whitespace around the operator, one operator kind per
call (chains like a - b - c fine, mixing refused), ^ takes exactly
two operands, and score/density each measure one slug at a time. The
expansion keeps a positive index-engaging leaf in front by
construction, so the NOT inside -/^ rides the residual. The same
booleans remain writable by hand (scry_recipe('hedging') AND NOT scry_recipe('certainty')), and the contrast ratio
countIf(scry_recipe('a')) / countIf(scry_recipe('b')) per cohort
cancels base rates. A composition worth reusing gets published as its
own recipe (derived_from naming the algebra) — that also makes it
scoreable. Terms may carry form: "regex" (RE2, compiled to
match()): give a regex-bearing recipe token or phrase recall leaves
beside the patterns or it evaluates as a scan. Disjointness of two
instruments is a property to measure, not assume: countIf( scry_recipe('a & b')) beside each count says how much they overlap on
the relation you quantify over, and a stance pair that overlaps heavily
is one recipe with a missing stance.
Guiding knobs beyond the query text: snippet_chars (64-1200, default
240) widens each result's served context window; max_per_source (>=1)
caps any one source's share of the page; limit, sources, kinds,
from/to bound the pool. In-query, NEAR/50 sets the proximity window
in characters, "phrase"~3 the slop window in words, and word~1 the
edit-distance window for typo tolerance.
Any community- or venue-scoped question starts from an enumerated source
set: run the inexpensive partition-enumeration query on the candidate relations
(e.g. SELECT source, count() FROM forums.posts GROUP BY source; subreddit
and list catalogs likewise) and report which sources were consulted and
which excluded. Missing a source that was one GROUP BY away is the
corpus's most common research failure.
For multi-step research — several hypotheses, several sources, or any ask
where missing vocabulary would silently distort the answer — follow
references.md § Deep research operations: fan out lexical probes, keep a probe
ledger, verify the written report against the ledger, and end in a durable
artifact. Surface selection starts with schema: the compact catalog plus
per-relation stats is the shortlist; enumerate partition values yourself
rather than delegating the plan.
For any study that compares cohorts or tests a hypothesis (who does X more,
does trait A predict behavior B), follow references.md § Comparative study design before
writing the first query: pre-state the refuter, audit selection–outcome
independence, and climb no higher on the interpretation ladder than the
instrument licenses.
For academic work — finding papers, tracing citation neighborhoods, and
above all reviewer discovery — follow references.md § Academic papers and reviewer discovery.
Reviewer discovery is a coverage problem: enumerate every candidate pool
with its denominator, keep a candidate ledger, screen conflicts, rank on
explicit axes, and stop on pool exhaustion, never on "enough names."
Conduct
Every claim ships with its source row or it does not ship. Prefer the
denominator: report what was searched — relations, sources, probe terms —
not only what was found. When sources conflict, resolve the conflict or
report it; never average it away. Small bounded probes cast wide before
expensive queries close. Done means the written answer is checked against
the queries that actually ran.
Fixpoint programs (recursive graph search)
WITH RECURSIVE is served on /v1/scry/query (body must be anchor UNION ALL step; read the CTE only in the step's FROM/JOIN, never in a
subquery) — but every iteration rescans the joined relation
(~1.8 s per step on openalex.works), so declare x-scry-max-seconds. For
frontier-pruned walks — citation closures, filtered multi-hop expansions,
walked sets ranked semantically — send a program instead of SQL: POST /v1/scry/query with a JSON body
{"program": {...}} (MCP datalog).
A sql atom is one statement (LIMIT <= 50000, the relation cap): alone in its body it
seeds a set from column id; after a rel it hydrates that relation —
the rows it returns keep their parent/depth and gain the other
columns as attrs (the statement must read the relation: WHERE <key> IN {name} — the keys are hn_id, post_key, tweet_id, and the OpenAlex id URL).
Inside a sql atom, {name} binds an already-evaluated relation as a query-scoped table of its ids (OpenAlex ids retain their full URLs), bounded by the 50k relation cap.
Walk then hydrate:
{
"relations": {
"seed": {"bodies": [[{"sql": "SELECT hn_id AS id FROM hackernews.items WHERE scry_lex('claude code') AND hn_type = 'story' ORDER BY original_timestamp DESC LIMIT 100"}]]},
"thread": {"bodies": [[{"rel": "seed"}], [{"rel": "thread"}, {"edge": "hackernews.children"}]]},
"final": {"bodies": [[{"rel": "thread"}, {"sql": "SELECT hn_id AS id, original_author, left(payload, 200) AS text FROM hackernews.items WHERE hn_id IN {thread} LIMIT 500"}]]}
},
"out": ["final"],
"depth": 2
}
Aggregate the same thread by replacing final with the following definition (ids are handles, with kind absent unless an edge consumes them):
{"bodies": [[{"sql": "SELECT original_author AS id, count() AS replies FROM hackernews.items WHERE hn_id IN {thread} GROUP BY id ORDER BY replies DESC LIMIT 50"}]]}
A sql seed runs as your own statement, so seed from keyed reads; for an
account's tweets, use twitter.tweets_of from its account id instead of
filtering twitter.tweets by author_id.
A program is named relations
(sets of node ids) built from a closed atom vocabulary — ids seeds,
ann (top-k probe from an embed handle), rel (a body naming its own
relation recurses), edge (graph steps: OpenAlex references/cited_by;
twitter twitter.replies/twitter.quotes + inverses; hackernews.children/
parent/story_items; forums.children/parent/thread — and pivots
that change what a node is: openalex.authors/institutions/works_of,
twitter.by/following/followers/tweets_of, hackernews.by/items_of,
forums.by/posts_of, github.repos_of, bluesky.by/posts_of,
youtube.uploader/commenters, tiktok.videos_of, instagram.posts_of,
crawl.urls_of; rows carry kind; an unknown edge name returns the
catalog with measured costs), filter
(in-walk attribute prune — changes what gets expanded and billed), in
(intersection), not_in (stratified negation; on a recursive body it
prunes the walk itself) — plus an optional per-relation "rank": {handle, k} ordering final rows by exact distance to a handle
(OpenAlex only); a relation left out of out ships only its per-depth
counts, zero egress (out: [] is the census). Every evaluation step
is one ordinary metered statement under your own key; depth (default 3)
and 50k-row caps bound the walk; the envelope returns {id, kind, parent, depth} provenance rows (a sql atom's other columns ride in attrs), counts for every relation (an empty seed set
shows counts.seed.rows = 0), a meter with per_statement, and
truncations[] (empty = fixpoint over the graph the index holds). Prefer rank over intersecting a walk with a global ANN
top-k — measured near-empty overlap at corpus scale. Rank is terminal: it orders a relation's final rows
after the walk, so put it on the last relation (the hydrating one), not on a set another relation reads.
Bound bodies give a relation tuples and variables: declare "vars": ["S", "W"] and every body opens with a driving {"rel": {"name": "seed", "vars": ["S"]}} (naming its own relation recurses), then up to four {"edge": {"name": "references", "vars": ["S", "W"]}} joins whose source var is
already bound, {"rel": {name, vars}} joins and {"not": {name, vars}}
anti-joins against evaluated relations, and filters either on the var an
edge produces ({"filter": {"on": "W", "col": "publication_year", "op": ">=", "val": 2020}}) or between two vars ({"filter": {"on": "B", "op": "!=", "var": "A"}}). Every head/negated/filtered var needs an earlier
positive binding; kinds come from edges, not sql; cited_by goes last;
legacy atoms consume only unary bound relations. Rows return as {tuple, parent, depth} plus an envelope schemas map. A k-edge chain nests its
prefilters (three HN edges in one body read ~88M rows), so keep bodies to
one or two edges when intermediate sets are large. Coauthors in one step:
{"relations": {"a": {"bodies": [[{"ids": ["A5000000036"]}]]},
"co": {"vars": ["B"], "bodies": [[{"rel": {"name": "a", "vars": ["A"]}}, {"edge": {"name": "openalex.works_of", "vars": ["A", "W"]}}, {"edge": {"name": "openalex.authors", "vars": ["W", "B"]}}, {"filter": {"on": "B", "op": "!=", "var": "A"}}]]}},
"out": ["co"]}
The MCP tool contract carries ten worked templates, including a seed-keyed
citation closure and an anti-join.
Lexical range
Embeddings are for missing vocabulary. When you know the words — names,
handles, idioms, error strings, catchphrases — token search composed with
plain SQL is sharper and faster, and it composes further: GROUP BY, joins,
and window functions turn retrieval into measurement. The corpus is a
programmable instrument; the searches worth running are the ones only you
would think to compose. Shapes that reward that creativity:
- Earliest attestation:
hasToken(search_text_lc, 'term') on
internet.text ordered by original_timestamp ASC — when and where a
phrase first appeared.
- An author's written history: one handle across reddit, HN, and mailing
lists over two decades (
original_author on internet.text), drift
measured with countIf per year.
- Co-occurrence archaeology:
hasAllTokens with two rare tokens and a
date bound — who put two ideas together first.
- Relations as instruments: citation neighborhoods (
openalex.works),
cross-platform identity (persons.links; enterprise access), thread
structure (reddit.comments joined via link_id) — walkable graphs beside the
text.
Diversity
An ask for diverse, varied, unexpected, or orthogonal sources,
communities, angles, hypotheses, or probe phrasings — or simply more
creative — is a coverage problem, not a writing problem. A list written
in one breath anchors on its own first items, and a tuned model's first
items are the mode; temperature does not repair that, and neither does
asking yourself to be creative. Change the ask instead (references.md
§ Orthogonal enumeration carries the procedure and the SQL):
- Roster before imagination. Where the space is a measured value
space — forum
source, subreddits, stackexchange site, relation on
internet.text, package ecosystem — the diverse set is the roster
covered, not recalled: one GROUP BY enumerates it, choose across it,
and report what was left out.
- Field before list. Where the space is open — angles, registers,
hypotheses, communities no column names — write 3–6 axes that change
the mechanism of a candidate (venue family, era, stance, register,
scale, inversion), 2–6 values each, and cover the cells. One candidate
per cell, written from that cell's conjunction alone, before looking at
the others. The grid is the denominator — but only for independent
shots (one fresh context per cell, or the endpoint below): a single
context walking the cells is a list, and a list reports no coverage.
- Entropy from outside the model. You cannot make a random choice;
the corpus can.
ORDER BY rand() over a roster, rand() % over an
axis to draw cells, a seeded cityHash64 for a reproducible
permutation — take order and seeds from a query, never from your own
preference.
- Outsized fan-out is an endpoint.
POST /v1/creativity/outsized
{"brief": "...", "shots": 4..24} (MCP creativity) runs the
whole campaign server-side — an explicit possibility space, server
entropy, one fresh small-model context per cell, an
enumeration-before-proposal gate, one consolidation pass — and returns
field (the independent candidates) and a nugget. Brief it for
directions, not answers: "enumerate orthogonal source families /
probe phrasings / hypotheses for X, each with the community that would
hold it and the words it would use" — then run each direction as a
bounded count-first probe. Pass "field": "inquiry" for research
briefs (the default artifact bank is for deliverables; the
measurement is in references.md § The outsized endpoint). The roster
and field steps remain the primary instrument; the endpoint is the
wide net behind them. Wallet-funded, about two minutes, experimental.
Saturation sweeps
When the ask is exhaustive — find everything, leave nothing unturned —
the opening frame becomes a stopping rule and the enumeration discipline
above becomes its instrument.
- The grid is relations × vocabularies × time windows: shortlist every
plausibly-holding relation from the schema index, fan each concept
into its namings (practitioner jargon, plain speech,
adjacent-community dialect, era-bound terms), and track cells — an
unprobed cell is an open claim, not a conclusion.
- Run the lexical and semantic arms in parallel; they miss differently.
Chase edges — authors, threads, citations — with
datalog; batch
probes 16 per round trip.
- Stop at saturation, not satisfaction: the tenth probe is where a field
opens, and done is when new probes return only known rows. Report the
grid itself — probed, found, unprobed — not only the hits. The MCP
exhaustive_search prompt carries this frame for any MCP client.
Registered surfaces
The live schema is the coverage authority: relation inventory, row counts,
per-source composition, freshness, and coverage extents come from
GET /v1/scry/schema and each query response's coverage block, never from
static text. Every relation carries a discovery tier: the default schema
document serves full contracts for the ~two dozen primary-tier doors (one
start-here relation per corpus family) plus a compact depth_relations index
of every supporting table — users, edges, comment variants, per-corpus
embeddings — all equally queryable. ?relation=<names> fetches any full
contract, ?mode=index the whole catalog one line per relation, ?mode=full
the complete document. The doors:
| Door |
Purpose |
internet.text |
The unified lexical surface: one row per text document across every text relation (reddit, twitter, hackernews, stackexchange, mastodon, crawl, internet documents, academic, forums, mailing lists, bluesky, commoncrawl) with token-indexed search_text_lc — start corpus-wide lexical questions here; relation names the underlying surface for hydration |
academic.catalog |
One merged bibliographic row per paper across the whole academic estate; joins full text (academic.papers), assessments, and embeddings via paper_key |
openalex.works |
Scholarly work metadata, authorships, topics, citation graph |
books.catalog |
Unified bibliographic catalog (file-backed book index, DOI journal index, library metadata records); idx names the record family — see its value space |
embeddings.chunks |
The unified ANN vector surface over every embedded corpus |
twitter.tweets |
The historical Twitter archive |
reddit.posts |
Full-retention Reddit submissions; comments (reddit.comments, depth) join via link_id = concat('t3_', id) |
hackernews.items |
Hacker News items with source identity and timestamps |
stackexchange.posts |
Stack Exchange Q&A across landed sites (site value space is the roster) |
crawl.pages |
Promoted text extractions of crawled web pages — the live house-crawl corpus |
commoncrawl.distillate |
Clean genre-classified Common Crawl reading layer; CDX census and raw WET recall are its depth companions |
social.posts |
Six frozen fringe-platform archives (voat, parler, gab, telegram, discord, truth_social) as one relation — always filter platform; profiles/edges/community directories are its depth companions (social.users/edges/communities) |
github.repos |
The public GitHub repository universe (408M origins, Software Heritage export) keyed by owner; repo READMEs/docs/source live in github.documents (depth) |
packages.catalog |
One merged row per software package across ~36 registries (ecosystem value space is the roster) |
markets.catalog |
One folded row per prediction market across Kalshi, Polymarket, Manifold (source/status value spaces) |
judgements.scores_current |
Latest cardinal judgement score per lens, axis, and entity |
persons.links |
Cross-platform person resolution: public accounts clustered into persons by shared strong identity keys — enterprise relation, served to operator-approved accounts only (hello@scry.io); the persons.link_coverage/content_coverage aggregates stay open |
events.records |
In-person-event corpus (conferences), JSON records keyed by event_slug |
courts.china_judgments |
China Judgments Online archive: ~85M published judgments 1985–2021, Chinese full text + structured metadata |
cn_enterprise.companies |
China enterprise registry (GSXT), one best row per company keyed by USCC |
mailing_lists.messages |
Mailing-list and Usenet archive messages; the per-list roster is mailing_lists.catalog (depth) |
internet_archive.items |
Internet Archive item-catalog metadata (identifier, creator, mediatype, collection, ...) |
threads.posts |
Threads (Meta) public posts from the anonymous breadth crawl, 2023-05 onward; threads.profiles is the author roster |
vk.posts / vk.comments |
VK community wall posts and comments, 2007 onward, full-text indexed on lower(text); vk.communities is the roster |
nostr.events |
Nostr relay events (signed event JSON; kind 1 notes, 0 profiles) |
youtube.videos_live |
YouTube metadata as currently observed (1B+ videos since 2026-08) — youtube.videos is the frozen 2021 census |
wikipedia.articles |
English Wikipedia article text, full page set kept current by recentchanges; wikimedia.events is the recent-change event stream |
huggingface.repositories |
Hugging Face hub models/datasets/spaces with counters; huggingface.snapshots_daily is the daily history; huggingface.repo_details carries per-repo bytes on the hub (usedStorage), file sizes, and model details |
reddit.subreddits |
Subreddit directory (description, subscribers, type, flags); reddit.subreddit_rules / reddit.subreddit_wikis are its depth |
irs.form990 / cms.open_payments / cfpb.complaints / jobs.postings / legistar.matters |
Envelope relations (payload.record is the upstream record): nonprofit filings, industry-to-provider payments, consumer finance complaints, live ATS job postings, municipal legislative matters |
epstein.artifacts |
Source-native Epstein artifact index across DOJ and other public releases |
agents.skills |
Parsed SKILL.md documents from public agent-skill repositories |
lexicons.entries |
English lexicon envelopes: Wiktionary (kaikki.org) and GCIDE/Webster 1913 |
amazon.reviews / amazon.items |
Amazon Reviews 2023 (McAuley Lab): 571.5M product reviews 1996–2023 with full-text text, and the 48.2M-item catalog; join on parent_asin |
orkut.topics / orkut.replies |
Orkut community forums 2004–2014 from the Wayback Machine: 120.6M topics, 897.3M replies (body full-text indexed), mostly Brazilian Portuguese |
community_notes.notes / community_notes.ratings |
X Community Notes public export (2025-02-22): every note with its tweet_id, every rating; community_notes.status_history / community_notes.enrollment are depth |
twitter.recsys_follow_graph |
Twitter's RecSys 2022 follow graph, 261M anonymised edges — structure only, never joins twitter.users |
streams.vod_chat / streams.vods |
Replayed Twitch and Kick VOD chat (offset, user name, message) with the VOD roster; live Twitch IRC with ids is twitch.messages |
Schema contracts carry measured value_spaces — the live vocabulary of
categorical spine columns (forum source, stackexchange site, market
source/status, package ecosystem, book idx/content_type, tweet
lang, subreddits) with row counts. Read them before writing a WHERE on a
categorical column; never guess an enum value —
subreddit = 'MachineLearning' vs 'machinelearning' is the classic
silent zero.
Confirm enablement and columns with /v1/scry/schema. A relation omitted from
that response is unavailable, even if this skill names its family. A relation
the schema lists can still refuse at admission for your key; treat a refusal
as unavailable and use other relations. Never infer a table from a source
name.
Starter
set -a
_scry_env="${XDG_CONFIG_HOME:-$HOME/.config}/scry/env"
[ -f "$_scry_env" ] && . "$_scry_env"
[ ! -f "$_scry_env" ] && [ -f "$HOME/.scry/.env" ] && . "$HOME/.scry/.env"
unset _scry_env
set +a
curl -s https://api.scry.io/v1/scry/schema \
-H "Authorization: Bearer $SCRY_API_KEY"
curl -s https://api.scry.io/v1/scry/query \
-H "Authorization: Bearer $SCRY_API_KEY" \
-H "Content-Type: text/plain" \
--data "SELECT hn_id, title, original_author, original_timestamp, uri FROM hackernews.items WHERE title != '' ORDER BY original_timestamp DESC LIMIT 20"
Query permalinks
- Typed placeholders make a query repeatable. Put
{name:Type} in the SQL
and send each value as a URL argument:
POST /v1/scry/query?param_author=karpathy with
... WHERE original_author = {author:String} ... LIMIT 50. Approved
types: String, UInt8..UInt64, Int8..Int64, Float32, Float64,
Date, DateTime. Keep LIMIT literal.
- Backslashes in
String parameter values: ClickHouse parses the value
in its escaped format, so a raw \b becomes a backspace byte and a
regex such as \bRust\b matches nothing. Double each backslash
(\\bRust\\b) or write the regex without backslashes
((^|[^a-z])Rust($|[^a-z])). Inline string literals in the SQL body
already use literal escaping and do not have this problem.
- To keep a query, create a share:
POST /v1/scry/shares (MCP
share) with
{title, kind: "query", payload: {sql, params: [{name, type, default}], snapshot: {...}}}. title is required, snapshot must be an object
(use {} when there is nothing to freeze), and each declared parameter
must have a default. The response's permalink field is the share's
page URL — cite it as served; share_slug is its tail.
- The share page at
https://scry.io/s/{slug} renders each
parameter as a live control and re-runs the query as the reader plays.
Optional per-parameter hints shape the controls: label, description,
placeholder, choices (a list of values or {value, label} objects —
renders as buttons), min/max/step (a numeric type with both bounds
renders as a slider), and widget
(segmented|slider|number|text|date|datetime) to override the choice.
The run endpoint ignores hints; only name, type, and default bind
values. A share with good hints is an instant playground — prefer one
bounded, hinted template over many near-duplicate saved queries.
- To run a saved query again:
POST /v1/scry/shares/{slug}/run?param_n=100
(MCP share_run)
or JSON body {"params":{"n":100}} (the body wins). The stored SQL goes
through the full metered pipeline as the caller: normal authentication,
validation, and billing. Values that are not supplied use the declared
defaults.
- A standing research question is a share too:
kind: "question" with
payload: {prompt, brief?, asked_in?} — prompt is the person's research
desire in their own words, verbatim (never paraphrased), brief is
markdown on how to attack it (relations, angles, what a good answer looks
like), asked_in the public URL where it was said. Any share of any kind
contributes to a question by setting top-level answers to the
question's slug at creation (immutable after); the question's page and
JSON (contributions) list every public contribution, and its markdown
twin (?format=md) carries the literal contribute call. The open index is
https://scry.io/s (GET /v1/scry/shares?kind=question, no credential).
When someone voices a research want, post it as a question and hand them
the permalink; when you finish a piece of work on one, publish the finding
as a contribution — a hinted query share is the best kind, because the
question's page then carries a live playground.
Adjacent runtime surfaces
- Account, settings, and market state: MCP
whoami,
GET /v1/scry/pricing,
GET /v1/scry/price, GET /v1/scry/price/history.
- Per-query charges arrive in the query response body:
burden_nanodollars
(the metered burden of your query) beside
spend_nanodollars (what you actually paid under the fairness charge
law — can exceed the raw meter for heavy identities), plus duration_ms, read_rows,
read_bytes, and record_id. billing_mode names the regime:
free_slack means authenticated queries settle at $0 while the system
has slack — spend=0 with a large burden is that policy working, not a
metering defect. Daily totals
come from GET /v1/scry/account (spend_today_usd, queries_today).
- Every query response carries a
coverage block: one entry per referenced
relation with its measured extent, declared known_holes, and
freshness_lag_seconds. Read it before you interpret an empty result.
Zero rows inside a measured extent with no known hole is meaningful
absence; zero rows outside it means the range is not landed. An empty
result also carries empty_result_note stating this rule. Parse it
precisely: known_holes: null means the hole registry was unreadable
(coverage-hole information is UNAVAILABLE — not "no holes"; that is
known_holes: []). If extent_error is present, the extent shown is
the last good measurement, not a live one — check extent.computed_at
and treat the extent as advisory until the error clears (the
empty_result_note text itself weakens in this state). Polling for
data that has not landed yet? extent.max tells you the corpus right
edge — poll the schema's lightweight coverage, not your full query.
extent.newest_event_at carries that same edge as a full UTC
timestamp — the newest landed entry's own event time. Precision
follows the extent column: second precision on scan-basis relations;
Date columns (and parts-basis date metadata) resolve to midnight, so
check extent.basis before reading the clock part as exact.
- Pricing is fair, not capped: charges engage only under measured
congestion, weighted by your own rolling-week usage. The full law —
rates, bands, and the operator's current price multiplier — is
published as
charge_law on GET /v1/scry/pricing. Off-peak
research costs least (slack is free).
- Never get surprised by a query: send
X-Scry-Max-Seconds: <n> to give
a query a hard execution deadline (the runtime kills it at n seconds
with a timeout error; you pay only for what ran). X-Scry-Budget: <nanodollars> is a runaway kill-switch, not a spend statement: while
the system has slack a query bills nothing, and the budget still binds the
raw machine meter — a small cap kills large scans that would have
charged nothing (a full-corpus scan can meter ~10^8 nanodollars).
Omit it unless you deliberately want that guard; state your real
deadline on every long query — it also sharpens
…(truncated)
1---2name: scry3description: Use Scry's read-only SQL research surface through /v1/scry/schema and /v1/scry/query. Use for bounded SQL over registered public-corpus relations, source provenance, and registered vector helpers. Also use when a research ask wants diverse, varied, or orthogonal sources, angles, hypotheses, or probe phrasings — the skill carries the enumeration discipline and the /v1/creativity/outsized fan-out.4---56# Scry Skill78Scry is read-only SQL (ClickHouse dialect) over registered public corpora9— Hacker News, Reddit, the Twitter archive, books, papers, forums, SEC10filings, the crawl — one call from a question to cited rows. Queries are11free while the system has slack: every response reports `billing_mode`12and `spend_nanodollars`, and the money arguments (`x-scry-budget`,13`x-scry-max-seconds`; MCP `budget_nanodollars`, `max_seconds`) are14ceilings you choose, never fees. Ask your wildest curiosity.1516Three one-call questions (`POST /v1/scry/query` with `Content-Type:17text/plain`, or the MCP `sql` tool). The first Hacker News item to18mention bitcoin:1920```sql21SELECT hn_id, original_author, original_timestamp, title22FROM hackernews.items23WHERE hasToken(search_text_lc, 'bitcoin')24ORDER BY original_timestamp ASC25LIMIT 526```2728Who said "vibe coding" before Karpathy:2930```sql31SELECT tweet_id, original_timestamp, text32FROM twitter.tweets33WHERE hasAllTokens(search_text_lc, ['vibe', 'coding'])34 AND positionCaseInsensitive(search_text_lc, 'vibe coding') > 035 AND original_timestamp < '2025-02-01'36ORDER BY original_timestamp ASC37LIMIT 538```3940Where Reddit talked bitcoin in 2013:4142```sql43SELECT subreddit, count() AS n44FROM reddit.comments_popular45WHERE created_utc >= '2013-01-01' AND created_utc < '2014-01-01'46 AND hasToken(search_text_lc, 'bitcoin')47GROUP BY subreddit48ORDER BY n DESC49LIMIT 1050```5152Every response carries `rows`, `read_rows`, `coverage`,53`deadline_partial`, `truncated`, and the meter (`burden_nanodollars` is54what the machine did, `spend_nanodollars` what you paid). A cut scan55(`deadline_partial: true`, or a deadline error) wants a rarer token, a56tighter WHERE or LIMIT, or a smaller sibling relation57(`reddit.comments_popular` beside `reddit.comments`, `x_open.tweets`58beside `twitter.tweets`); the `x-scry-explain: 1` header (MCP `explain:59true`) pre-flights a wide statement for free — the index analysis returns60and nothing runs.6162Search like the answer exists. It almost always does — under a63vocabulary, a venue, or an era you have not probed yet — so treat every64empty result as a wrong probe before treating it as an absence. You are65covering a space, not fetching an answer: fan vocabularies, sweep66relations, cross time windows, run lexical and semantic arms in67parallel, chase edges, and keep going past the first sufficient-looking68hit — the tenth probe is where a field opens. Done is saturation — new69probes returning only rows already seen — never satisfaction. Report70the space covered, not just the hits.7172The live schema is the contract; static relation lists are only73orientation.7475**Skill generation**: `2026082203`7677## Workflow78791. Load the durable key from `~/.config/scry/env` (legacy `~/.scry/.env`80 still honored). Context is readable without a81 credential; schema, stats, and queries require your key. When the Scry82 MCP server is connected (the ExoPriors/skills plugin wires83 `mcp.scry.io` on install), use its tools directly — the OAuth84 connection is the credential and no key file is needed; the key path85 below serves raw HTTP. If neither an MCP connection nor a key is86 available, stop before going further and direct the user to87 `https://scry.io/#console`.882. Call `GET /v1/scry/context?mode=agent&skill_generation=2026082203`.89 For worked, measured query shapes, `GET /v1/scry/examples?mode=index`90 (free, no key) lists the query-complexity tree one row per entry —91 every entry introduces exactly one construct atop its parent's, from92 selectivity probe to semantic ANN, each with its observed wall time and93 the byte size of its SQL. `?slug=<slug>` fetches one entry's problem,94 SQL, technique, and measurement; `?mode=tree` nests the taxonomy,95 `?mode=chains` lists root-to-leaf ladder walks; the bare route returns96 every entry in full (144 KB).973. Discover from the doors. The default `GET /v1/scry/schema` document98 already carries full contracts for the primary-tier doors plus a compact99 `depth_relations` index of every supporting table; fetch further full100 contracts with `GET /v1/scry/schema?relation=<name>[,<name>]`, or101 `?mode=index` for the whole catalog as one `relation | tier | extent |102 lag | purpose` line per relation (both also exposed as the MCP `schema`103 tool's `mode` and `relation` arguments; the MCP default is the index and104 `mode="contract"` carries the product contract, census, and live105 statistics). Use only106 relations and helper functions returned there, and read each relation's107 `query_guidance` block — `filter_columns_first`, `indexed_predicates`,108 `coverage_note` — before writing the first predicate: it names the109 indexed access paths. Never guess column names from memory of similar110 sources — a wrong column returns the relation's real column roster in111 the error, so one failed query self-corrects in one step; an unknown112 relation returns the nearest registered names.1134. Send one SQL statement to `POST /v1/scry/query` with114 `Content-Type: text/plain`.1155. Semantic search: mint a named query vector with `POST /v1/scry/embed`116 `{text, name}`, then use it as the unquoted `@name` inside117 `scry_vector_topk_distance`; full patterns are in118 `references.md` § Scry query patterns. Query text craft dominates119 every other parameter: embed answer-shaped, exuberant passages —120 the paragraph you hope to find — never keyword stubs, and fan out121 registers (`references.md` § Writing the query text). The same endpoint takes122 `{expression, name}` to compose stored handles (contrast axes,123 centroids, debiasing) into a new saved handle with diagnostics —124 see `references.md` § Composing embeddings into saved handles and125 the schema's `vector_recipes`. The ANN set is dynamic — a relation126 leaves it while its vector index re-materializes — and the schema names127 the live set: only surfaces with `serves_ann: true` accept ANN ranking128 (the rest still serve plain SQL). ANN queries must be standalone (no129 JOIN); hydrate companion text in a second query.1306. Keep every query bounded with `LIMIT`. Start at 20 and widen only after131 inspecting row shape, provenance, and source coverage.132 Token search speed is governed by the rarest token: in133 `hasToken`/`hasAllTokens` filters include at least one distinctive134 token (a name, identifier, or unusual word) — all-common-word token135 sets scan a large share of the table and run 30-60s. A slow query's136 response carries a `performance_note` naming the fix. For broad137 topical questions with only common words, use the embeddings helpers138 instead.1397. Parse results from `rows`, not a `data` key: each row is a plain JSON140 array with values in column order. A client that reads `data` sees141 false empty results.142143## Memory144145Scry hosts one cross-platform memory document per account146(MCP `memory`/`memory_write`):147markdown, default slug `main`, 64KB, shared by every agent and harness the148user connects. At session start read it alongside context (version 0 +149empty content = none yet). At session end, consolidate durable user150preferences — including what worked against Scry: relations, query151patterns, vector handles — back into it under a `## Scry usage` heading.152Writes are whole-document compare-and-swap on `if_version`; a 409 returns153the current head — merge into it and retry. Keep it compressed: the cap is154the decay function. If the document is empty and the user's local agent155memory holds durable preferences, you may offer — once, and only with the156user's explicit approval — to consolidate them into Scry memory so they157travel across platforms. Encrypted at rest server-side.158159Do not use engine catalogs, foreign-dialect casts or operators, compatibility160helpers, or a fallback corpus database. Do not invent relations. Pass a161search-grammar line as `q` to MCP `sql`; SQL remains the only read verb.162163The `q` search grammar speaks a full lexical language: bare words AND164together; `"exact phrase"`; `a OR b`; `-term` / `-"phrase"` exclusion;165`( )` grouping; `/pattern/` regex over full text (case-insensitive,166negatable; RE2 only — SQL rejects lookaround and backreferences rather than167counting a prefilter's superset. A positive literal or token anchors the168query; `rust /[0-9]+/` can use `rust` to bound the regex residual, while169bare `/[0-9]+/` is refused); `word*` wildcards; `word~1` fuzzy170(typo-tolerant: a 4-24 char word resolves against the corpus vocabulary171into its real one-edit word forms and searches as their OR —172`query_plan.clamped` echoes the forms chosen; bare `~` means `~1`,173larger asks clamp to 1 with a note); `"exact phrase"~3` slop174(phrase words in order, at most N intervening words between neighbors,175max 50); and176`a NEAR b` / `a NEAR/50 b` proximity (uppercase NEAR; matches both orders177within N characters, default 100, max 1000; operands may be words, quoted178phrases, `/regex/`, or `(x OR y)` groups). Substrings and CJK phrases can179use a sufficiently built n-gram index; read the relation's capabilities,180not a corpus-wide availability claim.181182MCP `sql` with `q` requires one registered `relation`, never `"*"`.183It returns ordinary SQL rows and the executed `compiled_sql`; it does not184silently weaken a zero-result query. Inspect that SQL before interpreting185membership. With `explain: true`, the statement is validated and its186ClickHouse index analysis is returned without executing the corpus query,187beside a `forecast` — `rows_est`, `bytes_est_uncompressed` and `seconds_est`188from the measured rows and bytes per granule and the measured scan rate,189`fits_max_seconds` against the deadline the call would run under, and190`faster` (sibling relation plus the rewritten statement) when it does not.191Request `prompts/get` with `name: "query_guide"` and `tool: "sql"` for192composition patterns and the current input schema.193194The compiler's internal plan distinguishes declared indexes from measured195coverage: zero-built word indexes do not establish pruning, and partial196coverage is not complete coverage. `EXPLAIN` is the actual plan evidence,197especially for views whose backing indexes are not mapped in discovery.198Use bounded, independently recorded queries to compare several relations;199the MCP SQL tool does not accept a multi-relation grammar sweep.200201The grammar is also a first-class SQL operand: inside any202`POST /v1/scry/query` statement, `scry_lex('<line>')` expands203server-side into exactly the predicate `sql` with `explain` would return for204the statement's one registered relation — so205`WHERE scry_lex('"scaling laws" -toy')`,206`countIf(scry_lex('/GPT-[0-9]/')) AS hits`, and GROUP-BY histograms207over a lexical cohort are plain SQL. An optional second argument pins208the text expression (`scry_lex('rust', title)`); an operator the209relation cannot express is a hard error, never a silent drop. At most 8210calls per statement; one registered relation per statement.211212## Lexical recipes213214Reuse shared term instruments with `scry_recipe('<slug>'[, text])` for215membership and `scry_recipe_score('<slug>'[, text])` for token-weighted216score. Use `scry_recipe_density('<slug>'[, text])` for weighted term217occurrences per 1,000 characters across token, phrase, and regex members.218Discover them with MCP `recipes`; publish a complete measured219version with `recipe_write` and the returned head version as220`if_version`. Derive candidates read-only with `recipe_derive`, then curate noise, measure the instrument, and publish through `recipe_write`. Write a recipe when you derived at least five surface forms,221or when a polarity instrument survives reading 20 matches per cohort.222Read those matches before publishing, keep provenance and measurements223with the terms, and treat the stance as part of the recipe's identity.224The seeded shelf and choosing guidance live in `references.md` § The225recipe shelf; the author/thread/time/graph quantifier shapes that226recipes plug into are `references.md` § The quantifier chain; the full227plane-by-plane operator map — quorum and frequency gates, named228quantifiers, Allen span relations, life-history regex, epistemic229operator families — is `references.md` § The operator space.230231Composing recipes has an operand: `scry_recipe('a - b')` difference,232`scry_recipe('a & b')` intersection, `scry_recipe('a ^ b')`233exclusive-or — whitespace around the operator, one operator kind per234call (chains like `a - b - c` fine, mixing refused), `^` takes exactly235two operands, and score/density each measure one slug at a time. The236expansion keeps a positive index-engaging leaf in front by237construction, so the `NOT` inside `-`/`^` rides the residual. The same238booleans remain writable by hand (`scry_recipe('hedging') AND NOT239scry_recipe('certainty')`), and the contrast ratio240`countIf(scry_recipe('a')) / countIf(scry_recipe('b'))` per cohort241cancels base rates. A composition worth reusing gets published as its242own recipe (`derived_from` naming the algebra) — that also makes it243scoreable. Terms may carry `form: "regex"` (RE2, compiled to244`match()`): give a regex-bearing recipe token or phrase recall leaves245beside the patterns or it evaluates as a scan. Disjointness of two246instruments is a property to measure, not assume: `countIf(247scry_recipe('a & b'))` beside each count says how much they overlap on248the relation you quantify over, and a stance pair that overlaps heavily249is one recipe with a missing stance.250251Guiding knobs beyond the query text: `snippet_chars` (64-1200, default252240) widens each result's served context window; `max_per_source` (>=1)253caps any one source's share of the page; `limit`, `sources`, `kinds`,254`from`/`to` bound the pool. In-query, `NEAR/50` sets the proximity window255in characters, `"phrase"~3` the slop window in words, and `word~1` the256edit-distance window for typo tolerance.257258Any community- or venue-scoped question starts from an enumerated source259set: run the inexpensive partition-enumeration query on the candidate relations260(e.g. `SELECT source, count() FROM forums.posts GROUP BY source`; subreddit261and list catalogs likewise) and report which sources were consulted and262which excluded. Missing a source that was one GROUP BY away is the263corpus's most common research failure.264265For multi-step research — several hypotheses, several sources, or any ask266where missing vocabulary would silently distort the answer — follow267`references.md` § Deep research operations: fan out lexical probes, keep a probe268ledger, verify the written report against the ledger, and end in a durable269artifact. Surface selection starts with `schema`: the compact catalog plus270per-relation stats is the shortlist; enumerate partition values yourself271rather than delegating the plan.272273For any study that compares cohorts or tests a hypothesis (who does X more,274does trait A predict behavior B), follow `references.md` § Comparative study design before275writing the first query: pre-state the refuter, audit selection–outcome276independence, and climb no higher on the interpretation ladder than the277instrument licenses.278279For academic work — finding papers, tracing citation neighborhoods, and280above all reviewer discovery — follow `references.md` § Academic papers and reviewer discovery.281Reviewer discovery is a coverage problem: enumerate every candidate pool282with its denominator, keep a candidate ledger, screen conflicts, rank on283explicit axes, and stop on pool exhaustion, never on "enough names."284285## Conduct286287Every claim ships with its source row or it does not ship. Prefer the288denominator: report what was searched — relations, sources, probe terms —289not only what was found. When sources conflict, resolve the conflict or290report it; never average it away. Small bounded probes cast wide before291expensive queries close. Done means the written answer is checked against292the queries that actually ran.293294## Fixpoint programs (recursive graph search)295296`WITH RECURSIVE` is served on `/v1/scry/query` (body must be `anchor UNION297ALL step`; read the CTE only in the step's FROM/JOIN, never in a298subquery) — but every iteration rescans the joined relation299(~1.8 s per step on openalex.works), so declare `x-scry-max-seconds`. For300frontier-pruned walks — citation closures, filtered multi-hop expansions,301walked sets ranked semantically — send a program instead of SQL: `POST /v1/scry/query` with a JSON body302`{"program": {...}}` (MCP `datalog`).303A `sql` atom is one statement (`LIMIT <= 50000`, the relation cap): alone in its body it304seeds a set from column `id`; after a `rel` it hydrates that relation —305the rows it returns keep their `parent`/`depth` and gain the other306columns as `attrs` (the statement must read the relation: `WHERE <key> IN {name}` — the keys are hn_id, post_key, tweet_id, and the OpenAlex id URL).307Inside a `sql` atom, `{name}` binds an already-evaluated relation as a query-scoped table of its ids (OpenAlex ids retain their full URLs), bounded by the 50k relation cap.308309Walk then hydrate:310311```json312{313 "relations": {314 "seed": {"bodies": [[{"sql": "SELECT hn_id AS id FROM hackernews.items WHERE scry_lex('claude code') AND hn_type = 'story' ORDER BY original_timestamp DESC LIMIT 100"}]]},315 "thread": {"bodies": [[{"rel": "seed"}], [{"rel": "thread"}, {"edge": "hackernews.children"}]]},316 "final": {"bodies": [[{"rel": "thread"}, {"sql": "SELECT hn_id AS id, original_author, left(payload, 200) AS text FROM hackernews.items WHERE hn_id IN {thread} LIMIT 500"}]]}317 },318 "out": ["final"],319 "depth": 2320}321```322323Aggregate the same thread by replacing `final` with the following definition (ids are handles, with `kind` absent unless an edge consumes them):324325```json326{"bodies": [[{"sql": "SELECT original_author AS id, count() AS replies FROM hackernews.items WHERE hn_id IN {thread} GROUP BY id ORDER BY replies DESC LIMIT 50"}]]}327```328329A sql seed runs as your own statement, so seed from keyed reads; for an330account's tweets, use `twitter.tweets_of` from its account id instead of331filtering `twitter.tweets` by `author_id`.332A program is named relations333(sets of node ids) built from a closed atom vocabulary — `ids` seeds,334`ann` (top-k probe from an embed handle), `rel` (a body naming its own335relation recurses), `edge` (graph steps: OpenAlex `references`/`cited_by`;336twitter `twitter.replies`/`twitter.quotes` + inverses; `hackernews.children`/337`parent`/`story_items`; `forums.children`/`parent`/`thread` — and pivots338that change what a node is: `openalex.authors`/`institutions`/`works_of`,339`twitter.by`/`following`/`followers`/`tweets_of`, `hackernews.by`/`items_of`,340`forums.by`/`posts_of`, `github.repos_of`, `bluesky.by`/`posts_of`,341`youtube.uploader`/`commenters`, `tiktok.videos_of`, `instagram.posts_of`,342`crawl.urls_of`; rows carry `kind`; an unknown edge name returns the343catalog with measured costs), `filter`344(in-walk attribute prune — changes what gets expanded and billed), `in`345(intersection), `not_in` (stratified negation; on a recursive body it346prunes the walk itself) — plus an optional per-relation `"rank":347{handle, k}` ordering final rows by exact distance to a handle348(OpenAlex only); a relation left out of `out` ships only its per-depth349counts, zero egress (`out: []` is the census). Every evaluation step350is one ordinary metered statement under your own key; `depth` (default 3)351and 50k-row caps bound the walk; the envelope returns `{id, kind, parent,352depth}` provenance rows (a sql atom's other columns ride in `attrs`), `counts` for every relation (an empty seed set353shows `counts.seed.rows = 0`), a `meter` with `per_statement`, and354`truncations[]` (empty = fixpoint over the graph the index holds). Prefer `rank` over intersecting a walk with a global ANN355top-k — measured near-empty overlap at corpus scale. Rank is terminal: it orders a relation's final rows356after the walk, so put it on the last relation (the hydrating one), not on a set another relation reads.357358Bound bodies give a relation tuples and variables: declare `"vars": ["S",359"W"]` and every body opens with a driving `{"rel": {"name": "seed", "vars":360["S"]}}` (naming its own relation recurses), then up to four `{"edge":361{"name": "references", "vars": ["S", "W"]}}` joins whose source var is362already bound, `{"rel": {name, vars}}` joins and `{"not": {name, vars}}`363anti-joins against evaluated relations, and filters either on the var an364edge produces (`{"filter": {"on": "W", "col": "publication_year", "op":365">=", "val": 2020}}`) or between two vars (`{"filter": {"on": "B", "op":366"!=", "var": "A"}}`). Every head/negated/filtered var needs an earlier367positive binding; kinds come from edges, not sql; `cited_by` goes last;368legacy atoms consume only unary bound relations. Rows return as `{tuple,369parent, depth}` plus an envelope `schemas` map. A k-edge chain nests its370prefilters (three HN edges in one body read ~88M rows), so keep bodies to371one or two edges when intermediate sets are large. Coauthors in one step:372373```json374{"relations": {"a": {"bodies": [[{"ids": ["A5000000036"]}]]},375 "co": {"vars": ["B"], "bodies": [[{"rel": {"name": "a", "vars": ["A"]}}, {"edge": {"name": "openalex.works_of", "vars": ["A", "W"]}}, {"edge": {"name": "openalex.authors", "vars": ["W", "B"]}}, {"filter": {"on": "B", "op": "!=", "var": "A"}}]]}},376 "out": ["co"]}377```378379The MCP tool contract carries ten worked templates, including a seed-keyed380citation closure and an anti-join.381382## Lexical range383384Embeddings are for missing vocabulary. When you know the words — names,385handles, idioms, error strings, catchphrases — token search composed with386plain SQL is sharper and faster, and it composes further: GROUP BY, joins,387and window functions turn retrieval into measurement. The corpus is a388programmable instrument; the searches worth running are the ones only you389would think to compose. Shapes that reward that creativity:390391- Earliest attestation: `hasToken(search_text_lc, 'term')` on392 `internet.text` ordered by `original_timestamp ASC` — when and where a393 phrase first appeared.394- An author's written history: one handle across reddit, HN, and mailing395 lists over two decades (`original_author` on `internet.text`), drift396 measured with `countIf` per year.397- Co-occurrence archaeology: `hasAllTokens` with two rare tokens and a398 date bound — who put two ideas together first.399- Relations as instruments: citation neighborhoods (`openalex.works`),400 cross-platform identity (`persons.links`; enterprise access), thread401 structure (`reddit.comments` joined via `link_id`) — walkable graphs beside the402 text.403404## Diversity405406An ask for *diverse*, *varied*, *unexpected*, or *orthogonal* sources,407communities, angles, hypotheses, or probe phrasings — or simply *more408creative* — is a coverage problem, not a writing problem. A list written409in one breath anchors on its own first items, and a tuned model's first410items are the mode; temperature does not repair that, and neither does411asking yourself to be creative. Change the ask instead (`references.md`412§ Orthogonal enumeration carries the procedure and the SQL):413414- **Roster before imagination.** Where the space is a measured value415 space — forum `source`, subreddits, stackexchange `site`, `relation` on416 `internet.text`, package `ecosystem` — the diverse set is the roster417 *covered*, not recalled: one GROUP BY enumerates it, choose across it,418 and report what was left out.419- **Field before list.** Where the space is open — angles, registers,420 hypotheses, communities no column names — write 3–6 axes that change421 the *mechanism* of a candidate (venue family, era, stance, register,422 scale, inversion), 2–6 values each, and cover the cells. One candidate423 per cell, written from that cell's conjunction alone, before looking at424 the others. The grid is the denominator — but only for independent425 shots (one fresh context per cell, or the endpoint below): a single426 context walking the cells is a list, and a list reports no coverage.427- **Entropy from outside the model.** You cannot make a random choice;428 the corpus can. `ORDER BY rand()` over a roster, `rand() %` over an429 axis to draw cells, a seeded `cityHash64` for a reproducible430 permutation — take order and seeds from a query, never from your own431 preference.432- **Outsized fan-out is an endpoint.** `POST /v1/creativity/outsized`433 `{"brief": "...", "shots": 4..24}` (MCP `creativity`) runs the434 whole campaign server-side — an explicit possibility space, server435 entropy, one fresh small-model context per cell, an436 enumeration-before-proposal gate, one consolidation pass — and returns437 `field` (the independent candidates) and a `nugget`. Brief it for438 *directions*, not answers: "enumerate orthogonal source families /439 probe phrasings / hypotheses for X, each with the community that would440 hold it and the words it would use" — then run each direction as a441 bounded count-first probe. Pass `"field": "inquiry"` for research442 briefs (the default `artifact` bank is for deliverables; the443 measurement is in `references.md` § The outsized endpoint). The roster444 and field steps remain the primary instrument; the endpoint is the445 wide net behind them. Wallet-funded, about two minutes, experimental.446447### Saturation sweeps448449When the ask is exhaustive — find *everything*, leave nothing unturned —450the opening frame becomes a stopping rule and the enumeration discipline451above becomes its instrument.452453- The grid is relations × vocabularies × time windows: shortlist every454 plausibly-holding relation from the schema index, fan each concept455 into its namings (practitioner jargon, plain speech,456 adjacent-community dialect, era-bound terms), and track cells — an457 unprobed cell is an open claim, not a conclusion.458- Run the lexical and semantic arms in parallel; they miss differently.459 Chase edges — authors, threads, citations — with `datalog`; batch460 probes 16 per round trip.461- Stop at saturation, not satisfaction: the tenth probe is where a field462 opens, and done is when new probes return only known rows. Report the463 grid itself — probed, found, unprobed — not only the hits. The MCP464 `exhaustive_search` prompt carries this frame for any MCP client.465466## Registered surfaces467468The live schema is the coverage authority: relation inventory, row counts,469per-source composition, freshness, and coverage extents come from470`GET /v1/scry/schema` and each query response's `coverage` block, never from471static text. Every relation carries a discovery `tier`: the default schema472document serves full contracts for the ~two dozen **primary-tier doors** (one473start-here relation per corpus family) plus a compact `depth_relations` index474of every supporting table — users, edges, comment variants, per-corpus475embeddings — all equally queryable. `?relation=<names>` fetches any full476contract, `?mode=index` the whole catalog one line per relation, `?mode=full`477the complete document. The doors:478479| Door | Purpose |480| --- | --- |481| `internet.text` | The unified lexical surface: one row per text document across every text relation (reddit, twitter, hackernews, stackexchange, mastodon, crawl, internet documents, academic, forums, mailing lists, bluesky, commoncrawl) with token-indexed `search_text_lc` — start corpus-wide lexical questions here; `relation` names the underlying surface for hydration |482| `academic.catalog` | One merged bibliographic row per paper across the whole academic estate; joins full text (`academic.papers`), assessments, and embeddings via `paper_key` |483| `openalex.works` | Scholarly work metadata, authorships, topics, citation graph |484| `books.catalog` | Unified bibliographic catalog (file-backed book index, DOI journal index, library metadata records); `idx` names the record family — see its value space |485| `embeddings.chunks` | The unified ANN vector surface over every embedded corpus |486| `twitter.tweets` | The historical Twitter archive |487| `reddit.posts` | Full-retention Reddit submissions; comments (`reddit.comments`, depth) join via `link_id = concat('t3_', id)` |488| `hackernews.items` | Hacker News items with source identity and timestamps |489| `stackexchange.posts` | Stack Exchange Q&A across landed sites (`site` value space is the roster) |490| `crawl.pages` | Promoted text extractions of crawled web pages — the live house-crawl corpus |491| `commoncrawl.distillate` | Clean genre-classified Common Crawl reading layer; CDX census and raw WET recall are its depth companions |492| `social.posts` | Six frozen fringe-platform archives (voat, parler, gab, telegram, discord, truth_social) as one relation — always filter `platform`; profiles/edges/community directories are its depth companions (`social.users`/`edges`/`communities`) |493| `github.repos` | The public GitHub repository universe (408M origins, Software Heritage export) keyed by owner; repo READMEs/docs/source live in `github.documents` (depth) |494| `packages.catalog` | One merged row per software package across ~36 registries (`ecosystem` value space is the roster) |495| `markets.catalog` | One folded row per prediction market across Kalshi, Polymarket, Manifold (`source`/`status` value spaces) |496| `judgements.scores_current` | Latest cardinal judgement score per lens, axis, and entity |497| `persons.links` | Cross-platform person resolution: public accounts clustered into persons by shared strong identity keys — enterprise relation, served to operator-approved accounts only (hello@scry.io); the `persons.link_coverage`/`content_coverage` aggregates stay open |498| `events.records` | In-person-event corpus (conferences), JSON records keyed by `event_slug` |499| `courts.china_judgments` | China Judgments Online archive: ~85M published judgments 1985–2021, Chinese full text + structured metadata |500| `cn_enterprise.companies` | China enterprise registry (GSXT), one best row per company keyed by USCC |501| `mailing_lists.messages` | Mailing-list and Usenet archive messages; the per-list roster is `mailing_lists.catalog` (depth) |502| `internet_archive.items` | Internet Archive item-catalog metadata (identifier, creator, mediatype, collection, ...) |503| `threads.posts` | Threads (Meta) public posts from the anonymous breadth crawl, 2023-05 onward; `threads.profiles` is the author roster |504| `vk.posts` / `vk.comments` | VK community wall posts and comments, 2007 onward, full-text indexed on `lower(text)`; `vk.communities` is the roster |505| `nostr.events` | Nostr relay events (signed event JSON; `kind` 1 notes, 0 profiles) |506| `youtube.videos_live` | YouTube metadata as currently observed (1B+ videos since 2026-08) — `youtube.videos` is the frozen 2021 census |507| `wikipedia.articles` | English Wikipedia article text, full page set kept current by recentchanges; `wikimedia.events` is the recent-change event stream |508| `huggingface.repositories` | Hugging Face hub models/datasets/spaces with counters; `huggingface.snapshots_daily` is the daily history; `huggingface.repo_details` carries per-repo bytes on the hub (usedStorage), file sizes, and model details |509| `reddit.subreddits` | Subreddit directory (description, subscribers, type, flags); `reddit.subreddit_rules` / `reddit.subreddit_wikis` are its depth |510| `irs.form990` / `cms.open_payments` / `cfpb.complaints` / `jobs.postings` / `legistar.matters` | Envelope relations (`payload.record` is the upstream record): nonprofit filings, industry-to-provider payments, consumer finance complaints, live ATS job postings, municipal legislative matters |511| `epstein.artifacts` | Source-native Epstein artifact index across DOJ and other public releases |512| `agents.skills` | Parsed SKILL.md documents from public agent-skill repositories |513| `lexicons.entries` | English lexicon envelopes: Wiktionary (kaikki.org) and GCIDE/Webster 1913 |514| `amazon.reviews` / `amazon.items` | Amazon Reviews 2023 (McAuley Lab): 571.5M product reviews 1996–2023 with full-text `text`, and the 48.2M-item catalog; join on `parent_asin` |515| `orkut.topics` / `orkut.replies` | Orkut community forums 2004–2014 from the Wayback Machine: 120.6M topics, 897.3M replies (`body` full-text indexed), mostly Brazilian Portuguese |516| `community_notes.notes` / `community_notes.ratings` | X Community Notes public export (2025-02-22): every note with its `tweet_id`, every rating; `community_notes.status_history` / `community_notes.enrollment` are depth |517| `twitter.recsys_follow_graph` | Twitter's RecSys 2022 follow graph, 261M anonymised edges — structure only, never joins `twitter.users` |518| `streams.vod_chat` / `streams.vods` | Replayed Twitch and Kick VOD chat (offset, user name, message) with the VOD roster; live Twitch IRC with ids is `twitch.messages` |519520Schema contracts carry measured `value_spaces` — the live vocabulary of521categorical spine columns (forum `source`, stackexchange `site`, market522`source`/`status`, package `ecosystem`, book `idx`/`content_type`, tweet523`lang`, subreddits) with row counts. Read them before writing a WHERE on a524categorical column; never guess an enum value —525`subreddit = 'MachineLearning'` vs `'machinelearning'` is the classic526silent zero.527528Confirm enablement and columns with `/v1/scry/schema`. A relation omitted from529that response is unavailable, even if this skill names its family. A relation530the schema lists can still refuse at admission for your key; treat a refusal531as unavailable and use other relations. Never infer a table from a source532name.533534## Starter535536```bash537set -a538_scry_env="${XDG_CONFIG_HOME:-$HOME/.config}/scry/env"539[ -f "$_scry_env" ] && . "$_scry_env"540[ ! -f "$_scry_env" ] && [ -f "$HOME/.scry/.env" ] && . "$HOME/.scry/.env"541unset _scry_env542set +a543544curl -s https://api.scry.io/v1/scry/schema \545 -H "Authorization: Bearer $SCRY_API_KEY"546547curl -s https://api.scry.io/v1/scry/query \548 -H "Authorization: Bearer $SCRY_API_KEY" \549 -H "Content-Type: text/plain" \550 --data "SELECT hn_id, title, original_author, original_timestamp, uri FROM hackernews.items WHERE title != '' ORDER BY original_timestamp DESC LIMIT 20"551```552553## Query permalinks554555- Typed placeholders make a query repeatable. Put `{name:Type}` in the SQL556 and send each value as a URL argument:557 `POST /v1/scry/query?param_author=karpathy` with558 `... WHERE original_author = {author:String} ... LIMIT 50`. Approved559 types: `String`, `UInt8..UInt64`, `Int8..Int64`, `Float32`, `Float64`,560 `Date`, `DateTime`. Keep `LIMIT` literal.561- Backslashes in `String` parameter values: ClickHouse parses the value562 in its escaped format, so a raw `\b` becomes a backspace byte and a563 regex such as `\bRust\b` matches nothing. Double each backslash564 (`\\bRust\\b`) or write the regex without backslashes565 (`(^|[^a-z])Rust($|[^a-z])`). Inline string literals in the SQL body566 already use literal escaping and do not have this problem.567- To keep a query, create a share: `POST /v1/scry/shares` (MCP568 `share`) with569 `{title, kind: "query", payload: {sql, params: [{name, type, default}],570 snapshot: {...}}}`. `title` is required, `snapshot` must be an object571 (use `{}` when there is nothing to freeze), and each declared parameter572 must have a default. The response's `permalink` field is the share's573 page URL — cite it as served; `share_slug` is its tail.574- The share page at `https://scry.io/s/{slug}` renders each575 parameter as a live control and re-runs the query as the reader plays.576 Optional per-parameter hints shape the controls: `label`, `description`,577 `placeholder`, `choices` (a list of values or `{value, label}` objects —578 renders as buttons), `min`/`max`/`step` (a numeric type with both bounds579 renders as a slider), and `widget`580 (`segmented|slider|number|text|date|datetime`) to override the choice.581 The run endpoint ignores hints; only `name`, `type`, and `default` bind582 values. A share with good hints is an instant playground — prefer one583 bounded, hinted template over many near-duplicate saved queries.584- To run a saved query again: `POST /v1/scry/shares/{slug}/run?param_n=100`585 (MCP `share_run`)586 or JSON body `{"params":{"n":100}}` (the body wins). The stored SQL goes587 through the full metered pipeline as the caller: normal authentication,588 validation, and billing. Values that are not supplied use the declared589 defaults.590- A standing research question is a share too: `kind: "question"` with591 `payload: {prompt, brief?, asked_in?}` — `prompt` is the person's research592 desire in their own words, verbatim (never paraphrased), `brief` is593 markdown on how to attack it (relations, angles, what a good answer looks594 like), `asked_in` the public URL where it was said. Any share of any kind595 contributes to a question by setting top-level `answers` to the596 question's slug at creation (immutable after); the question's page and597 JSON (`contributions`) list every public contribution, and its markdown598 twin (`?format=md`) carries the literal contribute call. The open index is599 `https://scry.io/s` (`GET /v1/scry/shares?kind=question`, no credential).600 When someone voices a research want, post it as a question and hand them601 the permalink; when you finish a piece of work on one, publish the finding602 as a contribution — a hinted query share is the best kind, because the603 question's page then carries a live playground.604605## Adjacent runtime surfaces606607- Account, settings, and market state: MCP `whoami`,608 `GET /v1/scry/pricing`,609 `GET /v1/scry/price`, `GET /v1/scry/price/history`.610- Per-query charges arrive in the query response body: `burden_nanodollars`611 (the metered burden of your query) beside612 `spend_nanodollars` (what you actually paid under the fairness charge613 law — can exceed the raw meter for heavy identities), plus `duration_ms`, `read_rows`,614 `read_bytes`, and `record_id`. `billing_mode` names the regime:615 `free_slack` means authenticated queries settle at $0 while the system616 has slack — spend=0 with a large burden is that policy working, not a617 metering defect. Daily totals618 come from `GET /v1/scry/account` (`spend_today_usd`, `queries_today`).619- Every query response carries a `coverage` block: one entry per referenced620 relation with its measured `extent`, declared `known_holes`, and621 `freshness_lag_seconds`. Read it before you interpret an empty result.622 Zero rows inside a measured extent with no known hole is meaningful623 absence; zero rows outside it means the range is not landed. An empty624 result also carries `empty_result_note` stating this rule. Parse it625 precisely: `known_holes: null` means the hole registry was unreadable626 (coverage-hole information is UNAVAILABLE — not "no holes"; that is627 `known_holes: []`). If `extent_error` is present, the extent shown is628 the last good measurement, not a live one — check `extent.computed_at`629 and treat the extent as advisory until the error clears (the630 `empty_result_note` text itself weakens in this state). Polling for631 data that has not landed yet? `extent.max` tells you the corpus right632 edge — poll the schema's lightweight coverage, not your full query.633 `extent.newest_event_at` carries that same edge as a full UTC634 timestamp — the newest landed entry's own event time. Precision635 follows the extent column: second precision on scan-basis relations;636 Date columns (and parts-basis date metadata) resolve to midnight, so637 check `extent.basis` before reading the clock part as exact.638- Pricing is fair, not capped: charges engage only under measured639 congestion, weighted by your own rolling-week usage. The full law —640 rates, bands, and the operator's current price multiplier — is641 published as `charge_law` on `GET /v1/scry/pricing`. Off-peak642 research costs least (slack is free).643- Never get surprised by a query: send `X-Scry-Max-Seconds: <n>` to give644 a query a hard execution deadline (the runtime kills it at n seconds645 with a timeout error; you pay only for what ran). `X-Scry-Budget:646 <nanodollars>` is a runaway kill-switch, not a spend statement: while647 the system has slack a query bills nothing, and the budget still binds the648 raw machine meter — a small cap kills large scans that would have649 charged nothing (a full-corpus scan can meter ~10^8 nanodollars).650 Omit it unless you deliberately want that guard; state your real651 deadline on every long query — it also sharpens652653…(truncated)