Data Source Catalog
Source Selection Principles
- Free/open sources first, paid only if critical gaps remain
- Highest data quality sources first within each tier
- Respect rate limits -- never exceed published thresholds
- Diversify sources to enable cross-referencing (3+ per major claim)
Query Sequence by Pipeline
Person Pipeline (recommended order)
| Priority |
Source |
Cost |
What You Get |
How to Query |
| 1 |
Web search (name + company) |
Free |
Current role, recent news, public profiles |
WebSearch: "{name}" "{company}" site:linkedin.com OR site:github.com |
| 2 |
GitHub API |
Free (5K req/hr auth) |
Tech skills, repos, orgs, activity |
WebFetch: https://api.github.com/users/{handle} (if handle known) |
| 3 |
Semantic Scholar |
Free (10 req/sec auth) |
Publications, citations, h-index |
WebFetch: https://api.semanticscholar.org/graph/v1/author/search?query={name} |
| 4 |
USPTO PatentsView |
Free |
Patents, co-inventors |
WebFetch: https://api.patentsview.org/inventors/query?q={"inventor_last_name":"{last}"} |
| 5 |
YouTube Data API |
Free (10K units/day) |
Conference talks, presentations |
WebSearch: site:youtube.com "{name}" conference OR talk OR keynote |
| 6 |
Apollo.io |
Free tier (limited) |
Email, phone, enrichment |
WebFetch: people enrichment endpoint (requires API key) |
| 7 |
Hunter.io |
50 searches/mo free |
Email pattern, verification |
WebFetch: https://api.hunter.io/v2/email-finder?domain={domain}&first_name={first}&last_name={last} |
Company Pipeline (recommended order)
| Priority |
Source |
Cost |
What You Get |
How to Query |
| 1 |
Web search (company name) |
Free |
Overview, news, website |
WebSearch: "{company}" company overview funding technology |
| 2 |
Company website |
Free |
About, team, products, careers |
WebFetch: https://{domain}/about and /careers or /jobs |
| 3 |
OpenCorporates |
Free (open data) |
Registration, officers, filings |
WebSearch: site:opencorporates.com "{company}" |
| 4 |
SEC EDGAR |
Free, no auth |
10-K, 10-Q, proxy statements (US public) |
WebFetch: https://efts.sec.gov/LATEST/search-index?q=%22{company}%22&dateRange=custom&startdt=2025-01-01 |
| 5 |
Crunchbase |
Web free / API paid |
Funding, investors, acquisitions |
WebSearch: site:crunchbase.com "{company}" |
| 6 |
UK Companies House |
Free API |
UK company profiles, officers, filings |
WebFetch: https://api.company-information.service.gov.uk/search/companies?q={company} |
| 7 |
InfoCamere |
Paid (Italy) |
Italian registry, XBRL balance sheets |
WebSearch: site:registroimprese.it "{company}" |
| 8 |
Web search (tech stack) |
Free |
Technologies used |
WebSearch: "{company}" technology stack OR "built with" OR engineering blog |
| 9 |
Web search (job postings) |
Free |
Growth signals, tech needs, org structure |
WebSearch: "{company}" careers OR jobs site:linkedin.com OR site:greenhouse.io |
| 10 |
News search |
Free |
Recent developments, M&A, partnerships |
WebSearch: "{company}" news OR announcement OR funding 2026 |
Relationship Pipeline (recommended order)
| Priority |
Source |
What to Extract |
| 1 |
OpenCorporates officer search |
Board interlocks -- same person serving on multiple company boards |
| 2 |
SEC EDGAR proxy (DEF 14A) |
Board composition, compensation, related-party transactions |
| 3 |
Crunchbase investor data |
Shared investors between companies (warm intro paths) |
| 4 |
USPTO co-inventor data |
Technical collaboration networks |
| 5 |
Semantic Scholar co-author |
Research collaboration networks |
| 6 |
GitHub org membership |
Co-contribution networks in open source |
Source Details by Category
People Enrichment
Apollo.io: Free tier available | REST API | 275M+ contacts | Email: 1 credit, phone: 5-8 credits | 70-80% email success rate | Bulk enrichment: 10 records/call
Hunter.io: 50 searches/mo free | REST API | Domain search (all emails at domain) | Email finder (name+domain) | Email verifier | Single credit pool since 2025
ZoomInfo: Enterprise ($15K-30K/yr) | 100M+ profiles | Premium quality but out of scope for Phase 1
Company Registries
OpenCorporates: 200M+ companies globally | REST API v0.4.8 | Free for open data | Bellingcat-endorsed for investigative research
UK Companies House: Fully open data | REST + streaming API | Company profiles, officers, PSC (Persons with Significant Control)
InfoCamere (Italy): API available | XBRL balance sheets mandatory since 2010 | Paid per-query or subscription
SIRENE/INSEE (France): Free since 2017 | REST API | 30M+ establishments | SIREN/SIRET numbers
SEC EDGAR: Free, no auth | 10 req/sec | XBRL financial data | 30+ year history | Bulk ZIP archives nightly
Financial Data
SEC EDGAR: Gold standard for US public companies | data.sec.gov REST + bulk | Company facts, submissions, XBRL
Financial Modeling Prep: 250 req/day free | From $14/mo | Global financial data, ratios
Alpha Vantage: 25 calls/day free (severely limited) | From $49.99/mo | Fundamentals, technicals
Tech Stack Detection
BuiltWith: From $295/mo | 85K+ technologies | Historical data since 2007
Wappalyzer: 50 lookups/mo free | Broad categories | Real-time detection
DetectZeStack: From $15/mo | 7,200+ technologies | 60-90x cheaper than BuiltWith at volume
Practical: start with web search ("{company}" tech stack engineering blog), use Wappalyzer free tier for validation
Publications and Patents
Semantic Scholar: Free, no key required for basic | 200M+ papers | 1 req/sec (unauth), 10/sec (auth)
ORCID: Free | REST API v3.0 | Researcher IDs, affiliations, works
USPTO PatentsView: Free | US patents, inventors, co-inventor networks
EPO Open Patent Services: Free with registration | European patents | 10 req/sec
CrossRef: Free | DOI metadata, journal articles
News and Signals
NewsAPI.org: Free for dev (100 req/day, delayed) | 80K+ sources | Simple query syntax
NewsData.io: Free tier (200 credits/day) | 87K+ sources | 206 countries
GNews: Free (100 req/day) | Google News aggregation
RSS feeds: Free | Direct from outlets | Standard parsing
Social and Digital Footprint
GitHub API: Free 5K req/hr (auth) | Profiles, repos, contributions, orgs | Richest free OSINT source for technical people
Twitter/X API: Free tier nearly useless (1,500 tweets/mo) | Basic $100/mo | Not recommended for Phase 1
YouTube Data API: Free (10K units/day) | Conference talks, product demos
Key Limitations
- LinkedIn: Official API highly restricted. No open access to profile search. Third-party providers face legal risk (Proxycurl lawsuit Jan 2026). Use web search with
site:linkedin.com for basic profiles.
- EU private companies: Financial data significantly harder than US public. InfoCamere (Italy) and Companies House (UK) are best. Other EU countries vary.
- Speaking engagements: No aggregated API exists. Combine YouTube search + conference website scraping + Sessionize.
- Twitter/X: Post-acquisition API pricing makes it impractical for research at free/low-cost tier.
- WHOIS: Post-GDPR, most EU registrars redact personal data. Historical data (pre-2018) via SecurityTrails may still be valuable.
1---2name: nw-data-source-catalog3description: Complete catalog of OSINT data sources for business intelligence, organized by pipeline (people, company, relationship, digital footprint) with API details, pricing tiers, rate limits, and recommended query sequence.4---5
6# Data Source Catalog
7
8## Source Selection Principles
9
101. Free/open sources first, paid only if critical gaps remain
112. Highest data quality sources first within each tier
123. Respect rate limits -- never exceed published thresholds
134. Diversify sources to enable cross-referencing (3+ per major claim)
14
15## Query Sequence by Pipeline
16
17### Person Pipeline (recommended order)
18
19| Priority | Source | Cost | What You Get | How to Query |
20|----------|--------|------|-------------|-------------|
21| 1 | Web search (name + company) | Free | Current role, recent news, public profiles | WebSearch: `"{name}" "{company}" site:linkedin.com OR site:github.com` |
22| 2 | GitHub API | Free (5K req/hr auth) | Tech skills, repos, orgs, activity | WebFetch: `https://api.github.com/users/{handle}` (if handle known) |
23| 3 | Semantic Scholar | Free (10 req/sec auth) | Publications, citations, h-index | WebFetch: `https://api.semanticscholar.org/graph/v1/author/search?query={name}` |
24| 4 | USPTO PatentsView | Free | Patents, co-inventors | WebFetch: `https://api.patentsview.org/inventors/query?q={"inventor_last_name":"{last}"}` |
25| 5 | YouTube Data API | Free (10K units/day) | Conference talks, presentations | WebSearch: `site:youtube.com "{name}" conference OR talk OR keynote` |
26| 6 | Apollo.io | Free tier (limited) | Email, phone, enrichment | WebFetch: people enrichment endpoint (requires API key) |
27| 7 | Hunter.io | 50 searches/mo free | Email pattern, verification | WebFetch: `https://api.hunter.io/v2/email-finder?domain={domain}&first_name={first}&last_name={last}` |
28
29### Company Pipeline (recommended order)
30
31| Priority | Source | Cost | What You Get | How to Query |
32|----------|--------|------|-------------|-------------|
33| 1 | Web search (company name) | Free | Overview, news, website | WebSearch: `"{company}" company overview funding technology` |
34| 2 | Company website | Free | About, team, products, careers | WebFetch: `https://{domain}/about` and `/careers` or `/jobs` |
35| 3 | OpenCorporates | Free (open data) | Registration, officers, filings | WebSearch: `site:opencorporates.com "{company}"` |
36| 4 | SEC EDGAR | Free, no auth | 10-K, 10-Q, proxy statements (US public) | WebFetch: `https://efts.sec.gov/LATEST/search-index?q=%22{company}%22&dateRange=custom&startdt=2025-01-01` |
37| 5 | Crunchbase | Web free / API paid | Funding, investors, acquisitions | WebSearch: `site:crunchbase.com "{company}"` |
38| 6 | UK Companies House | Free API | UK company profiles, officers, filings | WebFetch: `https://api.company-information.service.gov.uk/search/companies?q={company}` |
39| 7 | InfoCamere | Paid (Italy) | Italian registry, XBRL balance sheets | WebSearch: `site:registroimprese.it "{company}"` |
40| 8 | Web search (tech stack) | Free | Technologies used | WebSearch: `"{company}" technology stack OR "built with" OR engineering blog` |
41| 9 | Web search (job postings) | Free | Growth signals, tech needs, org structure | WebSearch: `"{company}" careers OR jobs site:linkedin.com OR site:greenhouse.io` |
42| 10 | News search | Free | Recent developments, M&A, partnerships | WebSearch: `"{company}" news OR announcement OR funding 2026` |
43
44### Relationship Pipeline (recommended order)
45
46| Priority | Source | What to Extract |
47|----------|--------|----------------|
48| 1 | OpenCorporates officer search | Board interlocks -- same person serving on multiple company boards |
49| 2 | SEC EDGAR proxy (DEF 14A) | Board composition, compensation, related-party transactions |
50| 3 | Crunchbase investor data | Shared investors between companies (warm intro paths) |
51| 4 | USPTO co-inventor data | Technical collaboration networks |
52| 5 | Semantic Scholar co-author | Research collaboration networks |
53| 6 | GitHub org membership | Co-contribution networks in open source |
54
55## Source Details by Category
56
57### People Enrichment
58
59**Apollo.io**: Free tier available | REST API | 275M+ contacts | Email: 1 credit, phone: 5-8 credits | 70-80% email success rate | Bulk enrichment: 10 records/call
60**Hunter.io**: 50 searches/mo free | REST API | Domain search (all emails at domain) | Email finder (name+domain) | Email verifier | Single credit pool since 2025
61**ZoomInfo**: Enterprise ($15K-30K/yr) | 100M+ profiles | Premium quality but out of scope for Phase 1
62
63### Company Registries
64
65**OpenCorporates**: 200M+ companies globally | REST API v0.4.8 | Free for open data | Bellingcat-endorsed for investigative research
66**UK Companies House**: Fully open data | REST + streaming API | Company profiles, officers, PSC (Persons with Significant Control)
67**InfoCamere (Italy)**: API available | XBRL balance sheets mandatory since 2010 | Paid per-query or subscription
68**SIRENE/INSEE (France)**: Free since 2017 | REST API | 30M+ establishments | SIREN/SIRET numbers
69**SEC EDGAR**: Free, no auth | 10 req/sec | XBRL financial data | 30+ year history | Bulk ZIP archives nightly
70
71### Financial Data
72
73**SEC EDGAR**: Gold standard for US public companies | `data.sec.gov` REST + bulk | Company facts, submissions, XBRL
74**Financial Modeling Prep**: 250 req/day free | From $14/mo | Global financial data, ratios
75**Alpha Vantage**: 25 calls/day free (severely limited) | From $49.99/mo | Fundamentals, technicals
76
77### Tech Stack Detection
78
79**BuiltWith**: From $295/mo | 85K+ technologies | Historical data since 2007
80**Wappalyzer**: 50 lookups/mo free | Broad categories | Real-time detection
81**DetectZeStack**: From $15/mo | 7,200+ technologies | 60-90x cheaper than BuiltWith at volume
82Practical: start with web search (`"{company}" tech stack engineering blog`), use Wappalyzer free tier for validation
83
84### Publications and Patents
85
86**Semantic Scholar**: Free, no key required for basic | 200M+ papers | 1 req/sec (unauth), 10/sec (auth)
87**ORCID**: Free | REST API v3.0 | Researcher IDs, affiliations, works
88**USPTO PatentsView**: Free | US patents, inventors, co-inventor networks
89**EPO Open Patent Services**: Free with registration | European patents | 10 req/sec
90**CrossRef**: Free | DOI metadata, journal articles
91
92### News and Signals
93
94**NewsAPI.org**: Free for dev (100 req/day, delayed) | 80K+ sources | Simple query syntax
95**NewsData.io**: Free tier (200 credits/day) | 87K+ sources | 206 countries
96**GNews**: Free (100 req/day) | Google News aggregation
97**RSS feeds**: Free | Direct from outlets | Standard parsing
98
99### Social and Digital Footprint
100
101**GitHub API**: Free 5K req/hr (auth) | Profiles, repos, contributions, orgs | Richest free OSINT source for technical people
102**Twitter/X API**: Free tier nearly useless (1,500 tweets/mo) | Basic $100/mo | Not recommended for Phase 1
103**YouTube Data API**: Free (10K units/day) | Conference talks, product demos
104
105## Key Limitations
106
107- **LinkedIn**: Official API highly restricted. No open access to profile search. Third-party providers face legal risk (Proxycurl lawsuit Jan 2026). Use web search with `site:linkedin.com` for basic profiles.
108- **EU private companies**: Financial data significantly harder than US public. InfoCamere (Italy) and Companies House (UK) are best. Other EU countries vary.
109- **Speaking engagements**: No aggregated API exists. Combine YouTube search + conference website scraping + Sessionize.
110- **Twitter/X**: Post-acquisition API pricing makes it impractical for research at free/low-cost tier.
111- **WHOIS**: Post-GDPR, most EU registrars redact personal data. Historical data (pre-2018) via SecurityTrails may still be valuable.
majiayu000