npbuilds Alternative Data Monitoring — The Signal Layer
Alternative data is any dataset used for investment insight that is not traditional financial data (prices, earnings, filings). The industry has shifted from "alternative" to "essential" — an estimated 90% of institutional investors now use at least one alternative data source, up from 62% in 2023. Systematic integration of alt data has been associated with an estimated 3% annual return advantage for firms that use it well. The advantage accrues not from the data itself but from extracting ACTIONABLE SIGNALS before the market prices them in.
The Alt Data Taxonomy
What Makes Data "Alternative"
Traditional data: prices, volumes, financial statements, analyst estimates, economic indicators.
Alternative data: everything else that can inform investment decisions. The line blurs as "alternative" sources become mainstream — what was alternative in 2015 (satellite imagery) is table stakes for many hedge funds today.
The Alt Data Value Chain
Raw Data → Collection → Cleaning → Signal Extraction → Integration → Action
Each step has failure modes:
- Collection: Legal compliance, data quality, vendor reliability
- Cleaning: Missing data, outliers, changing formats, survivorship bias
- Signal Extraction: Overfitting, spurious correlation, look-ahead bias
- Integration: How to combine with fundamental analysis without double-counting
- Action: Position sizing, conviction weighting, signal decay management
Satellite Imagery
The original "alternative data" — seeing the physical world from orbit to infer economic activity.
Parking Lot Analysis
What it measures: Retail store traffic by counting cars in parking lots from satellite imagery.
Signal: More cars = more foot traffic = higher revenue. Compare period-over-period and year-over-year.
Application:
- Track Walmart, Target, Costco store traffic before earnings
- Monitor restaurant chains during new product launches
- Detect regional demand shifts (new store cannibalizing old store)
Providers: RS Metrics, Orbital Insight, Planet Labs
Limitations:
- Weather affects parking (snow covers lots, rain changes behavior)
- E-commerce shift reduces predictive power over time
- Cloud cover prevents satellite capture on some days
- Resolution: typically daily to weekly depending on satellite pass frequency
Oil Storage Monitoring
What it measures: Crude oil inventory levels by measuring the height of floating-roof storage tanks from satellite shadows.
Signal: Rising inventories = oversupply = bearish for oil prices. Falling inventories = tightening supply = bullish.
Application:
- Track Strategic Petroleum Reserve levels
- Monitor Chinese oil imports via coastal storage
- Verify (or contradict) official inventory reports
- Detect supply disruptions in real-time
Why it works: Official inventory data is reported weekly (EIA) or monthly. Satellite data is available within 24-72 hours of capture, giving a multi-day to multi-week information advantage.
Construction and Development Activity
What it measures: Physical construction progress — foundation laying, structural framing, project completion timelines.
Application:
- Real estate development tracking: Is the housing project on schedule?
- Factory construction: When will a new semiconductor fab come online?
- Infrastructure projects: Actual progress vs reported progress
- Disaster recovery: How quickly are damaged areas being rebuilt?
Agricultural Monitoring
What it measures: Crop health, acreage planted, growing conditions, water availability, deforestation.
Signal types:
- NDVI (Normalized Difference Vegetation Index): Plant health from spectral analysis
- Soil moisture levels from radar satellites
- Planted acreage vs USDA estimates
- Frost damage assessment within days of weather events
Application:
- Commodity trading: corn, soybean, wheat price forecasting
- Agribusiness equity research: input demand, yield estimates
- Climate risk assessment for agricultural regions
Delivery latency: Typically 24-72 hours from satellite capture to processed, actionable analysis. Some near-real-time services offer same-day delivery for premium clients.
Credit Card and Transaction Data
Aggregated, anonymized consumer spending data — one of the highest-signal alt data categories.
How It Works
- Payment processors and fintech companies aggregate transaction data
- Data is anonymized and aggregated (no individual cardholder data)
- Panels of millions of consumers provide statistically representative spending estimates
- Revenue estimates available days to weeks before official earnings reports
Signal Strength
Transaction data is one of the strongest predictors of company revenue because:
- It measures ACTUAL spending, not surveys or estimates
- It's available in near-real-time (1-3 day lag)
- It covers both online and in-store purchases
- Panel sizes of 5-10 million cardholders provide statistical reliability
Application Examples
- Retail: Track same-store sales growth for Target, Walmart, etc. before quarterly earnings
- Restaurants: Monitor average ticket size and visit frequency for Chipotle, McDonald's
- Subscriptions: Detect churn rates for Netflix, Spotify, gym memberships
- Travel: Airline and hotel spending as a real-time economic indicator
- Sector rotation: Aggregate spending categories to detect consumer behavior shifts
Providers
- Second Measure (acquired by Bloomberg): Wide merchant coverage, daily updates
- Bloomberg Transaction Data: Integrated into Bloomberg Terminal
- Earnest Research: Strong methodology, institutional focus
- Mastercard SpendingPulse: Broad coverage, macro-level insights
Privacy and Compliance Considerations
- GDPR (EU): Requires explicit consent for personal data processing. Aggregated, anonymized data may be exempt but legal grey areas exist.
- CCPA (California): Similar consent requirements. "Do Not Sell" provisions affect data collection.
- Panel bias: Transaction panels skew toward certain demographics, income levels, and geographies. Correct for this or your signal is biased.
- Vendor risk: If the data provider's collection method is later deemed non-compliant, your signal disappears.
Web Scraping and Digital Exhaust
Harvesting publicly available digital information for investment signals.
Product Pricing Data
What it tracks: Real-time pricing changes across e-commerce platforms and competitor websites.
Application:
- Detect pricing power: Is a company raising prices without losing volume?
- Monitor competitive dynamics: Who's discounting? Who's holding price?
- Inflation tracking: Build a real-time inflation index from millions of online prices (Billion Prices Project methodology)
- Input cost monitoring: Track raw material prices from wholesale platforms
Implementation: Automated scrapers that capture prices daily or hourly. Requires handling dynamic websites, anti-scraping measures, and data normalization across different formats.
Job Posting Analysis
What it tracks: Companies' hiring and firing patterns as a leading indicator of growth or contraction.
Signal logic:
- Increasing job postings in engineering/product = company is investing in growth
- Decreasing postings across all departments = potential cost-cutting or slowdown
- Shifts in job description language = strategic pivots (e.g., suddenly hiring AI/ML engineers)
- Geographic patterns = expansion plans, office openings/closings
Lag: Job postings lead actual hiring by 2-4 months, and hiring leads revenue impact by 3-6 months. This gives a 5-10 month forward view.
Data sources: Indeed, LinkedIn, Glassdoor, company career pages (scraped directly).
App Download and Usage Data
What it tracks: Mobile application download counts, daily active users (DAU), session length, retention.
Application:
- Track growth trajectory for mobile-first companies (Uber, DoorDash, dating apps)
- Detect user churn before it shows up in earnings
- Compare competitor apps' traction in real-time
- Monitor new market entry and geographic expansion
Providers: Sensor Tower, data.ai (formerly App Annie), Apptopia
Limitations: Download counts do not equal revenue. Usage data is more valuable than download data. iOS and Android panels may have different coverage quality.
Website Traffic Analysis
What it tracks: Unique visitors, page views, time on site, traffic sources for web-based businesses.
Application:
- E-commerce revenue estimation: Traffic x conversion rate x average order value
- Lead generation tracking for SaaS companies
- Content engagement for media and advertising businesses
- Competitive benchmarking: Who's gaining/losing share of web traffic?
Providers: SimilarWeb, Semrush, Ahrefs (primarily SEO-focused but useful for traffic estimates)
Google Trends
What it tracks: Relative search interest for any keyword over time and across geographies.
Application:
- Brand interest tracking: Is consumer awareness growing or declining?
- Product demand signals: Search spikes for "buy [product]" or "[product] review"
- Macro sentiment: Search volume for "recession," "unemployment," "inflation"
- Disease surveillance: Search patterns for symptoms predict outbreaks (Google Flu Trends concept)
Strengths: Free, real-time, global coverage, long history.
Limitations: Relative scale (not absolute), seasonal adjustment needed, search behavior changes over time (voice search, AI assistants alter patterns).
Social Media and NLP
Extracting investment signals from unstructured text at scale.
Twitter/X Financial Community
Signal types:
- Sentiment aggregation: Bullish vs bearish tone across financial Twitter
- Breaking news detection: Twitter often surfaces material events before news wires
- Retail positioning signals: What are individual investors excited about or scared of?
- Influencer tracking: Key accounts that move markets (Elon Musk effect)
Challenges: Bot contamination, pump-and-dump schemes, signal noise, data access restrictions (API pricing changes).
Reddit and Retail Investor Signals
Signal types:
- WallStreetBets mention frequency: What stocks are retail investors focused on?
- Sentiment scoring: Bullish/bearish/neutral classification of posts and comments
- "DD" (due diligence) post analysis: What theses are driving retail conviction?
- Short squeeze detection: Identifying heavily shorted stocks with rising retail interest
Application: As a CONTRARIAN indicator or early warning of retail-driven volatility. Not as a directional signal — retail sentiment is a poor predictor of returns beyond very short time frames.
Earnings Call NLP
What it analyzes: Tone, language patterns, and content of quarterly earnings conference calls.
Signal types:
- Tone analysis: Positive vs negative word frequency compared to prior calls
- Hedging language: Increased use of "may," "could," "uncertain" correlates with future disappointments
- Confidence metrics: Vocal analysis (pitch, pace, pauses) in live calls — experimental but promising
- Topic shifts: What topics are management emphasizing or avoiding compared to prior quarters?
- Q&A dynamics: How defensive or forthcoming is management when challenged by analysts?
Research finding: CEO linguistic tone changes predict future stock returns and earnings surprises. Deception detection models have shown modest but statistically significant predictive power.
News Sentiment Scoring
What it does: Automated scoring of news articles by topic, entity, and sentiment.
Approaches:
- Dictionary-based: Count positive/negative words (simple, fast, crude)
- Machine learning: Train classifiers on labeled financial news (more accurate, requires training data)
- Transformer-based: Use large language models fine-tuned on financial text (most accurate, most expensive)
FinBERT: A BERT model fine-tuned specifically on financial text. Outperforms general-purpose sentiment models on financial news, earnings calls, and analyst reports. Available open-source. Key capabilities:
- Financial sentiment classification (positive/negative/neutral)
- Aspect-based sentiment: What ASPECT of the company is the sentiment about?
- Domain-aware entity recognition: Distinguishes between Apple (company) and apple (fruit)
Aggregation and Alpha Decay
Social media signals face rapid alpha decay:
- When a signal is first discovered: maximum alpha
- As more investors use the same signal: alpha decays
- When the signal becomes consensus: alpha approaches zero or reverses
- Social media signals have among the FASTEST alpha decay of any alt data category — often weeks to months
Geolocation Data
Mobile device location data as a proxy for economic activity.
Foot Traffic Analysis
What it measures: Store visits, restaurant traffic, airport throughput, mall activity.
Application:
- Retail revenue estimation from foot traffic counts
- Restaurant same-store traffic trends
- Mall occupancy and retail health
- Airport passenger throughput as travel industry indicator
Providers: SafeGraph (now part of Dewey), Placer.ai, Foursquare
Key metric: Visits per location, normalized for seasonal patterns and weather effects.
Supply Chain Tracking
AIS (Automatic Identification System) data:
- Every commercial vessel broadcasts its position, speed, destination
- Track shipping volumes, port congestion, trade flow disruptions
- Detect supply chain bottlenecks before official data shows them
- Monitor oil tanker movements to estimate crude trade flows
Truck movement data:
- GPS tracking of commercial trucking fleets
- Freight volume as a leading indicator of economic activity
- Just-in-time supply chain disruption detection
- Cross-reference with rail and shipping for multimodal picture
Real-Time Economic Activity
Mobile phone movement as a macro indicator:
- Aggregate mobility data shows economic reopening/closing in real-time
- Commuting patterns indicate return-to-office trends and commercial real estate demand
- Consumer mobility to stores, restaurants, entertainment venues
- Cross-border movement for tourism and trade estimation
Note: Post-COVID, this data category has permanently expanded. Investors who tracked mobility data in March 2020 had a significant information advantage in predicting the economic impact timeline.
Government and Regulatory Data in Real-Time
Publicly available data from government sources, processed faster than most investors manage.
FRED (Federal Reserve Economic Data)
What it is: Over 800,000 economic data series from the Federal Reserve Bank of St. Louis.
Key series for investors:
- Yield curve data: 10Y-2Y spread (recession indicator)
- Inflation expectations: 5Y5Y forward breakeven inflation
- Labor market: Initial claims, continuing claims, JOLTS
- Financial conditions: Chicago Fed National Financial Conditions Index
- Money supply: M2, bank reserves, monetary base
- Housing: Case-Shiller, housing starts, mortgage rates
Advantage: Free API access, updated in real-time as releases occur. Most investors check FRED manually — automated monitoring gives you the data within seconds of release.
SEC EDGAR
What it is: All public filings with the Securities and Exchange Commission.
High-value signals:
- 13F filings: Quarterly holdings reports from institutional investors with >$100M AUM. Shows what smart money is buying and selling (with a 45-day lag).
- Form 4: Insider transactions (officer/director buys and sells). Insider buying clusters are among the strongest traditional signals.
- 8-K filings: Material event disclosures — earnings, M&A, executive changes. Real-time alert monitoring.
- 10-K/10-Q full text: Textual analysis of risk factor changes, MD&A language shifts, accounting policy modifications.
- SC 13D/13G: Activist investor position disclosures. Often move stock prices significantly.
Implementation: EDGAR full-text search API is free. Set up alerts for specific filers, filing types, or keywords.
Patent Filings
What it tracks: Innovation at the company level by monitoring patent applications and grants.
Signal logic:
- Patent velocity: Companies increasing patent filings are investing in R&D
- Patent quality: Citation-weighted patent scores correlate with future innovation output
- Technology overlap: Detect competitive encroachment before product announcements
- Talent mapping: Inventor names on patents reveal R&D team movements between companies
Sources: USPTO PAIR, Google Patents, Lens.org (all free).
Alt Data Integration Challenges
Signal Decay
The problem: As more investors use the same alternative dataset, the informational advantage (alpha) decays.
Lifecycle of an alt data signal:
- Discovery (few users): Maximum alpha, typically 3-5 years
- Early adoption (dozens of funds): Alpha begins to decay, 2-3 years
- Mainstream adoption (hundreds of funds): Alpha mostly gone, <1 year
- Commoditization (available to all): Alpha is zero or negative (crowded trade)
Implications:
- The most valuable alt data is the data nobody else is using
- By the time a dataset is discussed at conferences, its alpha is decaying
- Proprietary data collection (building your own scrapers, sensors) retains value longer
- The PROCESSING and interpretation of data matters more than access to the raw data
Overfitting Risk
The problem: Testing many datasets guarantees some will show spurious correlations with returns.
The numbers: If you test 100 uncorrelated signals at 95% confidence, you expect 5 to appear significant by pure chance.
Protection:
- Out-of-sample testing: Always reserve data for validation
- Multiple hypothesis correction: Apply Bonferroni or Benjamini-Hochberg adjustments
- Economic rationale: Only act on signals that have a LOGICAL reason to predict returns
- Walk-forward testing: Test on sequential time periods, not random splits
- Minimum sample size: Require at least 36 months of data before acting on a signal
Legal and Compliance Risk
The line between alternative data and insider trading:
Legal (generally):
- Satellite imagery of public spaces
- Web scraping of public websites (though ToS may restrict)
- Aggregated, anonymized transaction data
- Government filings and public records
- Social media posts
Grey area:
- Expert network calls (legal if compliant, but several insider trading cases have involved expert networks)
- Aggregated employee data (may reveal material non-public information about a company)
- Geolocation data with small sample sizes (may identify individuals)
Illegal:
- Data obtained by bribing insiders
- Hacked or stolen data (even if bought from a third party)
- Data that reveals material non-public information from a source with a duty to protect it
Rule of thumb: If a reasonable person would consider the data source "unfair," it probably crosses a line. When in doubt, consult compliance.
Cost Considerations
Premium alternative data pricing:
| Data Type | Annual Cost Range | Typical Users |
|---|---|---|
| Satellite imagery (full coverage) | $100K - $500K+ | Hedge funds, commodity traders |
| Transaction/credit card data | $200K - $1M+ | Fundamental equity funds |
| App usage/download data | $50K - $200K | Tech-focused funds |
| Web traffic data | $30K - $100K | E-commerce analysts |
| NLP/sentiment platforms | $50K - $300K | Quantitative funds |
| Geolocation/foot traffic | $50K - $200K | Real estate, retail funds |
| Custom data builds | $100K - $500K+ | Anyone with unique needs |
ROI calculation: If a dataset costs $200K/year and manages $100M AUM, it needs to generate 0.2% alpha to break even. This is achievable but NOT guaranteed.
Free and Accessible Alt Data Sources
Not all valuable alternative data costs six figures. Many of the highest-signal sources are free.
FRED (Free)
- 800,000+ economic time series
- API access with Python, R, Excel integration
- Updated in real-time as economic data releases occur
- Build automated alerts for yield curve changes, employment surprises, inflation data
OpenInsider (Free)
- Tracks SEC Form 4 insider transactions in a user-friendly format
- Filter by: company, insider role, transaction type, date range
- Cluster buys (multiple insiders buying within 2 weeks) are the strongest signal
- Insider selling is less informative (many legitimate reasons to sell)
Quiver Quantitative (Freemium)
- Government contracts: Which companies are winning federal spending
- Congressional trading: Stock trades by members of Congress (disclosed with a lag)
- Corporate lobbying expenditure
- Insider trading aggregation
- Some features free, advanced analytics require subscription
Google Trends (Free)
- Relative search interest for any keyword, any geography, any time period
- Compare up to 5 search terms simultaneously
- Download data in CSV format for quantitative analysis
- Category filtering to reduce noise (e.g., "Finance" category only)
- Useful for brand awareness, product demand, macro sentiment
CFTC Commitments of Traders Reports (Free)
- Weekly futures positioning data for commercial hedgers, large speculators, and small speculators
- Covers all major commodity, currency, and financial futures
- Extreme positioning often precedes reversals
- Available every Friday at 3:30 PM ET for positions as of the prior Tuesday
- Long history enables regime analysis and percentile rankings
SEC EDGAR Full-Text Search (Free)
- Search across all SEC filings by keyword, company, filing type, date range
- XBRL data for structured financial statement extraction
- RSS feeds for real-time filing alerts
- Build custom monitoring for specific companies, filing types, or keywords
- Python libraries (sec-edgar-downloader, edgartools) simplify access
Building a Monitoring Dashboard
A practical framework for integrating free alt data into a monitoring system.
Minimum Viable Dashboard
Layer 1 — Macro Regime (updated daily):
- FRED yield curve (10Y-2Y spread): recession probability
- FRED financial conditions index: stress indicator
- Google Trends "recession" search volume: retail fear gauge
- CFTC positioning extremes: crowded trade alerts
Layer 2 — Sector/Industry (updated weekly):
- Google Trends for key product/brand searches in your portfolio sectors
- Job posting trends for portfolio companies and competitors
- EDGAR 8-K alerts for material events in portfolio holdings
- Insider transaction monitoring via OpenInsider
Layer 3 — Position-Level (updated as available):
- EDGAR 13F tracking: What are top holders doing with your positions?
- Google Trends for company-specific brand interest
- App download/usage data for mobile-centric holdings
- News sentiment scoring for earnings and event-driven alerts
Dashboard Architecture
Data Sources → Collectors → Database → Signal Engine → Alert System → Review
Collectors: Python scripts (cron jobs or cloud functions) that pull data from APIs and websites on schedule.
Database: SQLite for small portfolios, PostgreSQL for larger setups. Store raw data AND processed signals.
Signal Engine: Rules-based alerts for threshold breaches. Start simple — complex ML models come later.
Alert System: Email, Slack, or SMS notifications when signals fire. Prioritize by signal strength and portfolio relevance.
Review: Weekly synthesis of all signals. Monthly deeper analysis of signal performance and false positive rates.
Signal Quality Assessment
Not all alt data signals are created equal. Score each on:
| Criterion | Low (1) | Medium (3) | High (5) |
|---|---|---|---|
| Economic rationale | No clear logic | Plausible story | Strong causal mechanism |
| Historical correlation | <0.1 | 0.1-0.3 | >0.3 |
| Lead time | Same-day or lagging | 1-4 weeks | 1+ months |
| Data quality | Frequent gaps/errors | Occasional issues | Consistently clean |
| Exclusivity | Everyone uses it | Moderate adoption | Few users |
| Cost | >$100K/year | $10K-$100K/year | Free or <$10K/year |
Decision rule: Prioritize signals scoring >20 out of 30. Signals scoring <12 are likely not worth the integration effort.
Common Pitfalls
- Data snooping: Testing dozens of alt data sources and reporting only the ones that "work" historically
- Ignoring base rates: Most stocks go up most of the time — your signal needs to beat this base rate
- Confusing correlation with causation: Satellite data of parking lots correlates with revenue because BOTH correlate with economic activity
- Overweighting novelty: New data feels more valuable than it is because of the novelty premium in attention
- Neglecting maintenance: Scrapers break, APIs change, data formats shift — alt data pipelines require ongoing engineering
- Ignoring signal decay: A signal that worked for 3 years may have already been discovered and arbitraged away
- Insufficient sample size: Drawing conclusions from 6 months of data when you need 36+ months for statistical reliability
Cross-Domain Connections
- Data-science/data-wrangling/feature-engineering: The alt data value chain (raw → cleaned → signal → integration) IS the feature engineering pipeline. Satellite imagery requires spatial features, NLP requires text embeddings (FinBERT), transaction data requires temporal aggregation — all feature engineering applied to financial signals.
- Data-science/statistical-analysis/statistical-testing: Signal validation requires hypothesis testing with multiple comparison correction. Testing dozens of alt data signals and reporting only the ones that "work" is textbook p-hacking — the statistical-testing skill provides the correction methods (Bonferroni, BH).
- Data-science/ml-engineering/drift-detection: Alt data signal decay is drift — a signal that worked for 3 years degrades as it's discovered and arbitraged. Monitoring signal quality over time uses drift detection methodology.
Related Skills
- sentiment-signals — Alternative data feeds (credit card, satellite, scrape) often serve as raw inputs for sentiment signal construction. Alt-data-monitoring is one input pipeline; sentiment-signals is one downstream framework that consumes it.