# Research Company > Capture a single company's website into the web-research store as a structured, cited dossier (store//profile.md) using Firecrawl. Use whenever the user wants to capture or re-capture a company from its website — "research company X", "/research-company acme.com", "capture acme into the store", "profile this competitor", "add X to the company store". One company at a time, keyed by canonical domain. It captures durable STATE (what the company is/sells/how it's positioned, from its own site) — not events (news/funding/M&A) or judgments (threat/fit/relevance), which belong to downstream consumers. NOT for answering questions from already-captured data — "tell me about X", "what does X charge" — that's /query-companies; if a warm fresh capture exists, this verb stops and hands off there instead of presenting the dossier. Only stale/new companies spend Firecrawl credits. - Skill: `brianbolze/research-company` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add brianbolze/research-company` - Raw SKILL.md: https://api.skillmd.com/api/skills/brianbolze/research-company/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: brianbolze (https://skillmd.com/u/brianbolze) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/brianbolze/research-company --- # /research-company — capture a company into the store Turn `research company X` into a cached, structured, cited dossier at `store//profile.md`. Firecrawl captures the site once; every later read filters that structure instead of re-scraping. **Capture mechanics are deterministic (the `fc.py` script bakes in the hazard knobs); the judgment — domain resolution, key-page selection, and the synthesis into `profile.md` — is yours.** ## Before you start — read the two authority docs (don't reinvent them) Two contracts govern this verb. The **capture playbook** ships *with* this skill (a sibling file); the **store contract** lives in the engine repo — resolve the engine root for that one: ```bash # engine root = the repo holding this skill (works through the ~/.claude/skills symlink): ENGINE="$(cd "$(dirname "$(realpath "$0")")/../.." 2>/dev/null && pwd)" # if $0 is unset, the skill dir is .../skills/research-company # canonical fallback (single-user setup): "/Users/brianbolze/Library/Mobile Documents/com~apple~CloudDocs/Web Research" ``` 1. **`firecrawl-capture.md`** (next to this file) — the capture mechanics. **§1 is the recipe; the `fc.py` calls below execute it.** §5 is the when-it-breaks lookup (geo-misroute, bot defense, map noise, SPA soft-404s) — skim it so you recognize a hazard when you hit one. 2. **`SCHEMA.md` + `TAXONOMIES.md`** (engine root) — how to write `profile.md`: the frontmatter fields, the closed value sets, the body sections. This is the enrichment contract (step 7). Read it before writing. `scripts/fc.py` (next to this file) is the workhorse: it auto-reads `FIRECRAWL_API_KEY`, applies `maxAge:0` + `location:US` + `waitFor` + the all-formats bundle, persists raw JSON + cleaned md + screenshots to `store//captures//`, and logs a manifest for the verify step. **Always capture through `fc.py`** — never hand-roll curl scrapes (you'll drop a hazard knob). > **Interpreter note.** If a subshell (often a `for` loop) reports `python3: command not found`, the pyenv shim isn't on its PATH — resolve once with `PY=$(command -v python3 || echo /opt/homebrew/bin/python3)` and call `"$PY" scripts/fc.py …`. fc.py is stdlib-only, so any Python 3 works. ## The capture loop Run in order. Steps 3–6 are the credit spend (~7–10 credits for a clean basic run = 1 map + 1 homepage + 5–8 key pages); 1–2 and 7–8 are free. Overages are add-ons, not the homepage pass (all-formats rides one credit) — enhanced-proxy retries (+4), PDF pages (+1/pg), and re-scrapes each show up per-call in `fc.py spend`. **One phase per turn.** Never issue a step in the same message as the step that consumes its output — a big interdependent batch returns no intermediate state, and that vacuum is where fabricated "results" get narrated before any tool returns (the openai.com run invented a block, a fallback, and prices this way). Independent calls (map + homepage) still batch fine; the rule is about *dependency*, not parallelism. **0. Pre-flight (free).** `python3 scripts/fc.py credits` — note `remainingCredits`. If low (≲20), warn the user before spending. This global balance is **headroom only** — it's a shared key, so don't diff it for this run's cost; that number comes from `fc.py spend` at step 8, summed from each call's own billed credits. (Key check is automatic; `fc.py` aborts if it can't find the key.) **1. Resolve the domain → slug (free, and it decides the store key).** `curl -sIL https://` and see what it *resolves to*. The final canonical host is the **store folder slug** (dots→dashes, e.g. `honehealth.com` → `honehealth-com`) **and** the `domain:` field — but **strip a bare leading `www.`** (it's a canonicalizing redirect, not a meaningful subdomain), so `www.maximustribe.com` → slug `maximustribe-com`, `domain: maximustribe.com`. Keep meaningful subdomains (`aws.amazon.com` → `aws-amazon-com`). The domain you were handed, if different, goes in `aliases:`. If it resolves to a *different live company*, flag a collision and stop to confirm — don't key on it. (A 403/429 here just means bot-defended; proceed to Firecrawl.) Once the slug is known, stamp the run clock before any work this verb may perform: ```bash RUN_STARTED_AT="$(python3 "$WEB_RESEARCH_HOME/scripts/runrecord.py" now)" ``` Warm-skip runs write **no** run record; this clock is only carried forward if the run actually writes an artifact. (Contract: [`RUNS.md`](../../modules/RUNS.md).) **2. Seed from the store + freshness gate (free).** If `store//profile.md` exists: - Read its `site_notes` (the capture playbook for *this* site — inherit it, don't rediscover), `key_pages`, and `captured_at` - Move the previous capture into `captures/_archive/` so that the most recent capture is always obvious and the captures folder doesn’t look massive. - **Coarse freshness:** if `captured_at` is recent (< ~7 days) and the user didn't ask for a refresh, **stop without capturing and hand off** — report the warm status (per-layer clocks from `store.py find`) and route the answer through `/query-companies`. This verb decides capture-vs-skip; presenting the dossier is the consume verb's job, never this one's. Otherwise re-capture (a fresh `captures//` folder; the old one is preserved). *(On a bare/guided invocation the skip-vs-refresh call surfaces at **step 2.5** instead of auto-deciding here.)* **2.5. Guided pre-flight — one question batch, then go (free, conditional).** Two on-ramps into the spend: - **Express** — the invocation already carried intent (a focus, a `refresh`, a module ask, or a plain "just go"). Honor it and skip this step. - **Guided** — a bare invocation (`/research-company acme.com`, nothing else). Before spending a credit, surface the run as **one** `AskUserQuestion` batch (never drip questions across turns), every option defaulted so the user can glance-and-accept. Shape it from what steps 1–2 found: - **This run** *(include only when a warm < ~7-day capture exists)* — `Skip — capture is warm` (default; the answer routes through `/query-companies`) · `Re-capture fresh`. The freshness gate's skip-vs-refresh call made explicit — the one place a human override beats the silent auto-skip. Stale/new → nothing warm; omit this question. - **Output scope** *(modules combine — multi-select)* — `Standard profile` (default — already includes the `logos:{}` multi-ratio wordmark/logomark/og marks, gathered on every capture; [§1.2](firecrawl-capture.md)) · `+ per-SKU offerings.md` (the Tier-1 roster; [§1.1](firecrawl-capture.md)) · `+ offerings.md with flagship product images` (also pull each flagship's clean **hero product render** for a design / rendering-reference consumer — rides the same PDP capture, stored at `captures//images/.`; [§1.1](firecrawl-capture.md)) · `+ telehealth.md cohort pack` (for a telehealth company — the 8 vertical classification cuts the universal profile can't tell apart; near-free, rides the profile pages; contract [`TELEHEALTH.md`](../../modules/cohort-packs/TELEHEALTH.md), lint `cohortcheck.py`) · `+ productivity_saas.md cohort pack` (for a horizontal productivity / work-software company — the 6 cuts the universal profile reads identical; near-free, rides the profile pages; contract [`PRODUCTIVITY_SAAS.md`](../../modules/cohort-packs/PRODUCTIVITY_SAAS.md), lint `cohortcheck.py --cohort productivity_saas`). Don't offer a depth dial — page count already flexes with `portfolio_shape`; the site decides depth better than a blind setting. **`offerings.md` is shape-gated:** if `offering_category` reads Services/Consulting, or the company has no enumerable priced SKU (bespoke / project work), **warn** that there's likely nothing to roster and default to skipping it with a recorded reason ([OFFERINGS "When to write it"](../../modules/OFFERINGS.md)). Warn, don't block — the category can resolve late or wrong. - **Emphasis** *(free-text, optional)* — "Anything to focus on or watch for?" It *biases* page selection (step 4), never *subtracts*: the core `profile.md` contract still gets filled, anything missed goes to `unverified_fields`, so a guided profile stays corpus-comparable. Branch on the answer: `Skip` → stop and hand off to `/query-companies` (~$0, no capture); otherwise carry scope + emphasis through steps 3–8. **A non-vanilla run leaves one trace.** If the guided run deviated from a plain capture — emphasis given, a module added (offerings), or a refresh forced over a still-warm capture — append a single **`Run profile:`** line to the Provenance section in step 8 (e.g. `Run profile: guided — emphasis "enterprise pricing"; +offerings`). A vanilla run adds nothing; clean profiles stay clean. *(Part of SCHEMA's fixed Provenance set as of `2.4`.)* **3. Map + homepage, together (2 credits).** Different endpoints — safe to run in one batch: ```bash python3 scripts/fc.py map https:// --slug --verb research-company python3 scripts/fc.py scrape https:// --slug --name homepage --homepage --verb research-company ``` `--verb research-company` tags each manifest line so `scripts/runcost.py` can attribute credit cost by verb (the `/deepen-offerings` preset passes its own — §below). `map` is subdomains-off by default — a clean marketing-host inventory (the docs/dev subdomain is dropped, §5.3). It's still a **sample** (big sites) or can be **empty** (custom SPAs), so homepage `links` are the reliable discovery surface. Pull both. **4. Pick key pages (judgment, free).** The map seeds candidates, but **select from homepage `links`** — the durable surface that also catches signal subdomains the map drops (`investors.`, `careers.`). **Only ever scrape a URL in the captured inventory — never hand-type a path from convention or prior knowledge** (`/about`, `/pricing`, `/ai`, …): a guessed path can return a real-sized 404 stub that poisons the profile (the Qualtrics run burned 4 credits this way; see BACKLOG "junk soft-404 stubs"). `verify` now fingerprints these (§5.6 junk soft-404 gate), but prevention is cheaper — it saves the wasted credit *and* the cleanup. Missing an expected page? Run `map --search ""` to surface it, don't guess. **Filter map noise first** (playbook §5.3): content/funnel paths (`/blog`, `/learning`, `/case-studies`, `/partner`, `/sweepstakes`, …) + locale prefixes (`/en-uk`, `/de-eu`, …). Then pick **~4-8 signal pages**: pricing, products/treatments, how-it-works, about — whatever carries the company's offering + model + claims. The about / company info / history page would also be helpful - whatever carries founding history, key metrics the company makes public, and key company events. **Homepage caveat:** if the homepage is an app shell / storefront / logged-in app — a marketplace, a big retailer, a SaaS that drops you straight into the product — it carries little positioning; there the about/company page *is* the primary self-description, so lean on it plus the category/product pages. Let the site's apparent breadth guide depth (a `Single`-shape brand needs fewer pages than a `Multi-product` one; see `portfolio_shape` in TAXONOMIES). **5. Scrape key pages — serially, no burst (1 credit each).** A parallel burst is what trips the geo/cache shell + 429s (§5.1/5.2). One at a time: ```bash python3 scripts/fc.py scrape https:/// --slug --name --verb research-company ``` Use a clear `--name` per page (`pricing`, `weight_loss`, `about`). Include linked PDFs if they add signal (pricing/spec sheets — 1 credit/page, prime primary source). **6. Verify scrapes (free).** `python3 scripts/fc.py verify --slug `. It checks sourceURL-match, md5-uniqueness across pages, **and a junk soft-404 gate**. A **DUP BODY** = §5.1 contamination (identical body for distinct URLs) — re-scrape the affected page, retrying once with `--proxy enhanced` (+4 credits) if a hard block. A **JUNK SOFT-404** = the page's title/heading declares itself "not found" (a dead/guessed path) — `rm` the flagged `.md` (the 404, if it's a finding, goes in prose). Don't discard a page on HTTP status alone — SPAs return 404 with full *correct* content (§5.6); trust the body, but a "Page Not Found" headline is the tell that it's junk, not content. (`verify` also lints `profile.md` once it exists — it just defers here, pre-write; you re-run it in step 7. It exits nonzero on any issue.) **7. Enrich → write `profile.md` (free, the valuable step).** Read the *whole* capture — every `captures//*.md`, the screenshots in `.payloads/*.png`, the homepage's `branding` payload, and its **structured layer** (`python3 scripts/fc.py signals --slug ` slices the `rawHtml` JSON-LD + `
`/`