Market Intel Research

Market Intel — Research Skill

Turn a vague "study this market" into a ranked, evidence-backed, actionable brief — and challenge it against your own last snapshot.

Ground rules (non-negotiable)

1. Public data only. Everything comes from legally scraped public sources (OSM, public directories, search). No login, no CAPTCHA bypass, no ToS violations. If a source requires credentials, stop and say so — don't improvise scraping.
2. Evidence over vibes. Every claim in a report carries a source (which crawl, which report, which page). No source = mark it as estimate.
3. Revenue/scale are PROXIES. Private micro-business revenue is not public. Use review counts × category benchmarks → band + confidence. Never fake precision. Say "estimated RM X–Y", not "RM X".
4. Free-first stack. Prefer $0 tools (OSM, browser automation) over paid scrapers. Only propose paid tools when the free path is genuinely blocked.
5. Respect the environment: the engine is Python; write temp scripts to a scratch dir rather than inline shell one-liners.

The pipeline (research → plan → spar)

Stage 1 — Crawl (who exists)

osm_crawl --area <area> [--category <vertical>]      # baseline POI density
gmaps_enrich --query "<vertical> <area>" --limit N   # phone/rating/reviews/website

• Free, keyless baselining first; enrichment second.
• Enrichment is fragile (layout changes): early-exit on 2 no-new-result cycles; incremental saves so timeouts don't lose data.

Stage 2 — Report (what the data says)

niche_report --area <area>          # per-niche opportunity scores 0-100
pitch_list --area <area>            # reachable businesses with NO website, ranked

• The grey line is the wedge: businesses that are active (reviews) but digitally invisible (no website). That gap = the sales opportunity.

Stage 3 — Suggest (the plan)

suggest --area <area> --tier flash|pro|ultra

• LLM stage ONLY where judgment is needed. Reads the niche report + pitch list → produces: niche ranking with willingness-to-pay, TOP N targets, pitch angles, wedges, risks, next steps.
• Terse, decisive output: "decisions and reasons", not hedging.

Stage 4 — Spar (challenge your own plan)

spar <vertical> --against <snapshot>

• Re-run against the last locked pipeline snapshot. Output: what changed, what's now uncontested, what's saturated, and a verdict (action/reject/ defer).

Stage 5 — Persist + query

pgvector_load --area <area>         # push to vector store for semantic queries

• Idempotent (dedupe by name+coords). Keeps its own table — never mixes into other tenants' corpora.

Output discipline

• Brief: STUDY_<vertical>_<area>_brief.md — problem, data, ranking, targets, wedge, next action.
• Spar: STUDY_<vertical>_spar.md — diff against snapshot + verdict.
• Both: recommendations are ranked by a score (0-100) that is traceable to the scoring function, and every target row has the raw evidence fields.

Known pitfalls (learned the hard way)

• Rating is not always in visible text — some locales put it in an aria-label attribute (5.0 / 5); parse both.
• Too-specific queries return 0 — a query phrased like " " can return nothing while "" works. Keep queries broad, filter locally.
• Lazy-loaded results — need explicit scroll of the results feed; pass element handles, not locators.
• Timeout safety — a full multi-vertical sweep exceeds most shell timeouts; run per-vertical and merge incrementally.
• Analysts bias: family-diverse models disagree on purpose — when the plan is built by a panel, record who said what, don't average to mush.

What to do when something's off

• Crawl empty → broaden query / check source reachability (the free source wants a User-Agent header).
• Report feels wrong → re-check scoring inputs, don't adjust the output.
• Panel disagreement → surface the disagreement as a decision, then decide with the evidence table — the report must say which evidence won.

Skill pattern: evidence → rank → plan → spar. The value is the discipline, not the scraping.