# Wsb Data Pipeline

> Scrape r/wallstreetbets posts for specific stock/forex tickers (TSLA, AAPL, XAUUSD, NVDA, GME) over historical date ranges and export Post Title, Date, Upvotes, Number of Comments, and Ticker Mentions to an Excel workbook. Use when the user asks to collect, update, or re-run WSB Reddit data for the herd-behavior research project, or asks about ticker date ranges, the data source, or output columns for that dataset.

- Skill: `faaz17/wsb-data-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add faaz17/wsb-data-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/faaz17/wsb-data-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: Faaz17 (https://skillmd.com/u/faaz17)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/faaz17/wsb-data-pipeline

---


# WSB Data Pipeline

Collects historical r/wallstreetbets post data per ticker into `output/wsb_ticker_data.xlsx` (one sheet per ticker) for the herd-behavior sentiment research (see `Research Points.docx`).

## Data source

Reddit's live API/search does not return multi-year-old posts reliably. This pipeline uses the **Arctic Shift API** (`https://arctic-shift.photon-reddit.com/api/posts/search`), a free public mirror of the Pushshift historical Reddit archive, instead. No Reddit credentials are required.

## Running the pipeline

Always run the bundled script — do not hand-write scraping/Excel code, the script already handles pagination, rate-limit backoff, caching, and deduping.

```bash
cd "<project root>"
.venv/Scripts/python.exe scripts/scrape_wsb.py                 # all 5 tickers, full configured ranges
.venv/Scripts/python.exe scripts/scrape_wsb.py --ticker GME     # one ticker only
.venv/Scripts/python.exe scripts/scrape_wsb.py --ticker TSLA --after 2024-01-01 --before 2024-02-01  # custom range
.venv/Scripts/python.exe scripts/scrape_wsb.py --no-cache       # force re-fetch, ignore cached API pages
```

On Windows PowerShell use `.\.venv\Scripts\python.exe` instead of the forward-slash form.

Each run merges into the existing `output/wsb_ticker_data.xlsx` (sheets for tickers not re-run are preserved), so a single ticker can be refreshed without re-scraping the rest.

## Configured ticker ranges (in `scripts/scrape_wsb.py`, `TICKERS` dict)

| Ticker | Aliases searched | Date range |
|---|---|---|
| TSLA | TSLA, Tesla | last 5 years (2021-06-30 to 2026-06-30) |
| AAPL | AAPL, Apple | last 3 years (2023-06-30 to 2026-06-30) |
| XAUUSD | XAUUSD, XAU/USD, Gold | last 3 years (2023-06-30 to 2026-06-30) |
| NVDA | NVDA, Nvidia, NVIDIA | last 4 years (2022-06-30 to 2026-06-30) |
| GME | GME, GameStop | 2020-01-01 to 2021-12-31 (fixed, per research scope) |

If the user wants different ranges or aliases, edit the `TICKERS` dict at the top of `scripts/scrape_wsb.py` rather than building one-off scripts.

## Output columns

`Post Title`, `Date` (UTC, ISO `YYYY-MM-DD`), `Upvotes` (Reddit score), `Number of Comments`, `Ticker Mentions` (regex word-boundary count of the ticker's aliases across title + selftext — use this as the per-post mention intensity, not a list of tickers).

## Things to watch for

- **Long runs**: TSLA/NVDA over multi-year windows can mean hundreds of API pages. Run tickers one at a time (`--ticker X`) if you want incremental progress/output rather than waiting for all 5.
- **Caching**: raw API pages are cached under `cache/<alias>/<after_ts>.json`. A re-run with the same date range is fast and free of API calls; use `--no-cache` only if the underlying data may have changed.
- **Rate limits**: the script checks the `X-Ratelimit-Remaining` response header and sleeps automatically when low — do not remove this or run multiple scrape processes in parallel against the same API.
- **XAUUSD is barely discussed on WSB** by exact ticker; the "Gold" alias is included to catch gold-related posts, but expect a much smaller dataset than the equities.
- After scraping, do not manually edit `output/wsb_ticker_data.xlsx` — re-run the script with updated ranges/aliases instead, so the file stays reproducible from `scripts/scrape_wsb.py`.