Algolia Crawler

Algolia Crawler

Turn web pages into a RAG-optimized Algolia index with the Algolia Crawler, driven entirely by the Algolia CLI (algolia crawler …). The Crawler visits URLs, extracts content with a JavaScript recordExtractor, and writes Algolia records on a schedule — no scraping code to maintain.

The single most important idea: crawling for RAG is not the same as crawling for site search. A RAG record is a chunk of text that will be dropped into an LLM's context window in isolation. That changes how you shape every record (see The RAG record mental model).

When to use this skill vs. the others

Rule of thumb: if the user wants to get web content into Algolia, you're in the right skill. Once the index exists and they want to ask questions over it, hand off to algobot-cli.

The RAG record mental model

Each record is retrieved and shown to an LLM alone, without the surrounding page. So optimize for that:

1. Small and single-topic. One idea, one fact, one Q&A per record — not a whole page. Granular records retrieve precisely and waste less of the context window.
2. Self-contained. Every record carries the context needed to stand alone: the entity it's about, the page title, the section heading. The LLM won't see the rest of the page.
3. Natural-language content, even for structured data. This is the highest-leverage move. Don't store only {model: "X", score: 0.87} — also synthesize a sentence: "Model X scores 0.87 on relevance…". Both keyword and vector retrieval, and the LLM itself, want prose.
4. Structured attributes for filtering. Alongside content, keep clean fields (category, type, date, numeric values) so retrieval can be scoped with filters and facets.

If you internalize only one thing from this skill, make it this list. Everything else is mechanics.

The workflow

Follow these steps in order. Full commands, config, and code live in the references — read them as you reach each step.

1. Set up the CLI and credentials. Install the CLI and export your Crawler credentials (a Crawler user ID + API key, distinct from your app keys), plus an Algolia write API key for the target index and your App ID. See cli.md.
2. Inspect the page first — is it JavaScript-rendered? Many modern pages load their real content via XHR after load, so a naive crawl indexes empty shells. Detect this and enable rendering. See javascript-rendering.md.
3. Write a recordExtractor that emits RAG records. For docs and prose, the recommended default is Algolia's Markdown helpers (helpers.markdown + helpers.splitTextIntoRecords) — they preserve headings/lists/code and are Algolia's own AskAI/RAG pattern. Hand-roll an extractor when you need per-entity structured records (tables, catalogs). Either way, follow the mental model above. See record-extractor.md.
4. Validate with algolia crawler test BEFORE indexing. It runs your config against a live URL and returns the records it would create, without writing anything. Use it as your feedback loop — and as a DOM inspector to fix selectors against the real rendered markup. See workflow.md.
5. Apply index settings explicitly. Do not rely on initialIndexSettings to configure the index — in practice it frequently does not apply. Set searchableAttributes, attributesForFaceting, etc. yourself with algolia settings import after the first crawl. See rag-index-settings.md.
6. Reindex, then verify by searching the index (algolia search) for a few realistic RAG questions before trusting it.
7. Schedule a recurring crawl so the index stays fresh as the source pages change.

Crawling a whole site or section (one URL → all its sub-pages)? The crawler discovers sub-pages via startUrls/sitemaps/discoveryPatterns — but the config must be shaped from all the page types it will hit, not just the start page. A config fitted to the landing page produces junk on the article/reference/blog templates it never saw. Discover the URL set, group it into templates, sample a representative page per template, and validate the extractor against each. See site-crawls.md.

The complete, copy-adaptable end-to-end run (create → test → settings → reindex → verify) is in workflow.md.

Two things that will bite you

These are the non-obvious failures. Both references cover them, but they're worth stating up front:

• JavaScript-rendered data. If the page shows Loading… placeholders in its initial HTML, or the numbers/table appear only after load, you must enable renderJavaScript. A crawl without it indexes nothing useful. (javascript-rendering.md)
• Values hidden in attributes. Rendered pages often keep the real value in an attribute (e.g. data-tip="Score: 79.8%") while the visible cell shows only a bar or a delta. Extract from the attribute, not the visible text. (record-extractor.md)

Driving it with the CLI

Everything runs through algolia crawler … (and algolia settings / algolia search for the index side). Full cheatsheet and the gotchas below are in cli.md.

algolia crawler create <name> -F config.json     # create (prints nothing on success)
algolia crawler test   <id> --url <url> [-F cfg] # extract records WITHOUT indexing
algolia crawler reindex <id>                      # start a crawl that writes records
algolia crawler get    <id>                        # inspect crawler + config/status
algolia crawler stats  <id>                        # crawl status summary

Two CLI realities to plan around (details in cli.md):

• Set renderJavaScript: true (boolean). The CLI only accepts the boolean form; the object form ({ enabled, patterns, waitTime }) makes get/list/create -F fail to parse. Boolean true uses the default render wait, which is enough for most pages — confirm with crawler test (empty values mean the page needs more render time).
• create prints nothing and there's no config-update or delete command. After create, recover the id from algolia crawler list (get needs a UUID — it doesn't accept a name). If list errors because another crawler in the account uses a non-boolean renderJavaScript, there's no pure-CLI recovery — look the id up via the Crawler REST list endpoint (GET /1/crawlers?name=<name>). To change a config, re-create; to delete, use the dashboard or REST DELETE.

References

• workflow.md — the full end-to-end CLI run: config, test-before-index loop, settings, reindex, verify, schedule.
• site-crawls.md — crawling a whole site/section: discover sub-pages, group page templates, sample and validate each, cover them with a branching extractor or multiple actions.
• record-extractor.md — RAG record patterns, helpers, reading values from attributes, chunking long prose.
• rag-index-settings.md — index settings tuned for retrieval, NeuralSearch/vector, query-time filtering for RAG.
• javascript-rendering.md — detecting and handling JavaScript-rendered pages.
• cli.md — setup, credentials, the algolia crawler command cheatsheet, and the CLI gotchas in detail.