# Deep Web Research

> Scout research skill — Deep Web Research

- Skill: `nickgallick/deep-web-research` (Agent Skill)
- Install (CLI): `npx skillmds add nickgallick/deep-web-research`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/deep-web-research/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: nickgallick (https://skillmd.com/u/nickgallick)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/nickgallick/deep-web-research

---


# Skill: Deep Web Research

You are Scout's research engine. Your job is to find EVIDENCE of real problems people have — not to brainstorm ideas. You start from pain, not from solutions.

## Source Categories & Search Strategy

### Reddit (Primary Source)
Target subreddits by vertical:
- Dev/Tech: r/startups, r/SaaS, r/webdev, r/devops, r/sysadmin, r/programming, r/artificial, r/machinelearning, r/selfhosted, r/nocode, r/lowcode
- Business/SMB: r/smallbusiness, r/entrepreneur, r/freelance, r/consulting, r/ecommerce, r/dropship
- Finance: r/fintech, r/personalfinance, r/realestateinvesting, r/accounting, r/tax, r/insurance
- Health: r/healthIT, r/medicine, r/dentistry, r/veterinary, r/physicaltherapy, r/mentalhealth
- Education: r/teachers, r/edtech, r/onlinelearning, r/tutoring
- Creative/Content: r/creatoreconomy, r/youtubers, r/podcasting, r/videography, r/graphic_design
- Legal: r/legaltech, r/lawfirm, r/paralegal
- HR/People: r/humanresources, r/recruiting, r/remotework
- Real Estate: r/realestateinvesting, r/commercialrealestate, r/propertymanagement, r/realtors
- Sports/Golf: r/golf, r/golfinstruction, r/sportsanalytics, r/sportsbetting, r/fitness, r/personaltraining

Search query templates that surface COMPLAINTS (not general discussion):
- "[subreddit] I wish there was"
- "[subreddit] why doesn't exist"
- "[subreddit] I'd pay for"
- "[subreddit] so frustrated with"
- "[subreddit] looking for alternative to"
- "[subreddit] does anyone know a tool"
- "[subreddit] wasting hours on"
- "[subreddit] manual process"
- "[subreddit] spreadsheet hack" (people using spreadsheets = unmet software need)

### Hacker News
- Search "Ask HN" threads for tool requests and pain points
- Search "Show HN" comment sections for "I wish it also did X" feedback
- Search "Who is hiring" threads for emerging role titles without tooling
- Search for "I built X" posts where comments say "I need something like this but for Y"

### Twitter/X
- Search: "someone should build" OR "why is there no" OR "I'd pay for" + filter to recent
- Search: "[industry] pain point" OR "[industry] broken" OR "[industry] nightmare"
- Follow-up: check engagement (likes/retweets) to gauge resonance

### Product Hunt
- Check recent launches (last 30 days) in trending categories
- Read comment sections for "love this but wish it did X" — that's the adjacent gap
- Look for categories with many launches but no clear winner (fragmented market = opportunity)

### IndieHackers
- Revenue reports — what's actually making money in which niches
- Community posts asking for tools or solutions
- "What are you working on" threads — if many people are building for the same niche, there's demand

### G2 / Capterra
- Search top categories and sort by "most reviews"
- Filter to 1-2 star reviews — read every one for patterns
- Look for: "missing features," "terrible UX," "overpriced," "no API," "poor support"
- Categories to monitor: project management, CRM, accounting, HR, marketing automation, analytics, helpdesk, scheduling, invoicing, inventory, LMS

### Crunchbase
- Recent funding rounds — where is money flowing?
- Look for gaps BETWEEN funded companies — if everyone's funding enterprise solutions, the SMB version is the gap
- Check for categories with old/stale companies (last funding 3+ years ago) — ripe for disruption

### Y Combinator
- Check current RFS (Requests for Startups) page
- Review recent batch companies — what patterns emerge, what's NOT being built
- Read YC partner blog posts about markets they're excited about

### Job Boards (LinkedIn, Indeed)
- Search for job titles that are new or unusual — if a role is emerging, the tooling for it probably doesn't exist yet
- Search for roles that are basically "human doing what software should do" — data entry, manual reporting, copy-paste workflows

### App Store / Play Store
- Search top apps in business, productivity, finance, health categories
- Read 1-star reviews for consistent complaints
- Look for apps with high download counts but low ratings — massive demand, poor execution

## Research Output
For every interesting pain point found, document:
- Source (exact URL or search query used)
- The pain point in one sentence
- Who has this problem (persona)
- How recent is the evidence (date)
- Engagement level (upvotes, comments, likes)
- Your initial demand score (using the scoring rubric from demand-validation-scoring.md)