Bggg Data Reddit

BGGG Reddit Data

Use Reddit's public JSON listings through a logged-in or public Chrome tab when direct shell access is blocked. Save every response before transforming it, then normalize locally with the bundled script.

VOC Project Layout（bggg 系列共用）

bggg VOC 系列 skill（bggg-data-amazon / bggg-data-reddit / bggg-data-x / bggg-voc-report / industry-orchestrator）共用一个项目文件夹，让多平台数据规整到同一处、下游分析零改路径。开工先确定项目根目录 <project>（用户指定，或新建 voc-<产品或主题slug>/），并从 <project> 根目录执行本 skill 的全部命令（下文相对路径都基于它）：

<project>/
  PROJECT.md            # 研究简报 + 决策日志（编排 skill 维护；单独使用可省）
  config/               # 采集目标：amazon_targets.tsv / reddit_queries.tsv / x_queries.tsv / keywords.txt
  work/<platform>/…     # 各平台原始证据、attempt 日志、request plan、manifest
  data/raw/             # 各平台规范化 JSONL（统一行契约，分析共用层）
  data/clean|coded/     # 下游清洗与编码（industry-orchestrator 维护）
  output/               # 报告与交付物（bggg-voc-report 写 output/report/）

本 skill 的落点：config/reddit_queries.tsv → work/reddit/（request plan 与 manifest）+ data/raw/source_json/（未修改源响应）→ data/raw/reddit_<lang>_<date>.jsonl。

Workflow

1. Define a small query file and build a request plan:

python3 scripts/build_request_plan.py \
  --queries config/reddit_queries.tsv \
  --output work/reddit/request_plan.json

Use tab-separated columns query, lang, and round. lang and round are optional.

2. Open https://www.reddit.com/ in Chrome. Fetch each planned URL from that Reddit page with same-origin fetch, including pagination through data.after. Do not read or export browser cookies, local storage, profile data, or credentials.

3. Save each unmodified response under data/raw/source_json/ and append one manifest row per saved file. Follow references/chrome_same_origin.md for the browser loop and manifest format.

4. Rank directly relevant parent posts by engagement and topical fit. Download complete comment trees for the highest-value parents with /comments/{post_id}.json?limit=500&depth=10&sort=top&raw_json=1. Treat replies as clustered under their parent thread; do not count thousands of replies from one viral thread as independent market prevalence.

5. Normalize and validate:

python3 scripts/normalize_reddit.py \
  --manifest work/reddit/source_manifest.jsonl \
  --output data/raw/reddit_EN_2026-07-25.jsonl \
  --summary work/reddit/normalize_summary.json

6. Report query count, parent-post count, comment count, unique rows, deleted/removed rows skipped, unresolved more nodes, HTTP failures, and parent-thread concentration.

Quality Rules

• Keep raw source JSON immutable. Write retries to new files rather than overwriting evidence.
• Use the Reddit fullname (t3_… or t1_…) as native_id; derive source_id with SHA-256.
• Preserve exact text_raw. Put translations or labels in downstream clean/coded files.
• Skip [deleted], [removed], empty bodies, ads, and promoted posts; count every exclusion.
• Search listings are discovery samples, not complete census data. Record sort, time window, query, page cursor, and collection time.
• Stop pagination only when after is null, the requested cap is reached, or repeated cursors/no-new-ID protection fires.
• Respect login walls, rate limits, and challenge pages. Back off and record the failure; never bypass authentication or change the user's account.
• Deduplicate exact native IDs first. Use similarity dedupe only in a later cleaning stage so raw evidence remains auditable.
• For prevalence statistics, cap or weight rows per parent thread and disclose the rule.

Degradation

If Chrome same-origin fetching is unavailable, try a normal public Reddit JSON request with a conservative user agent. If it is blocked, deliver the request plan and collection manifest with the exact failure; do not substitute search-engine snippets for full Reddit comments.

Resources

• scripts/build_request_plan.py: create deterministic search request and manifest templates.
• scripts/normalize_reddit.py: parse saved search listings and nested comment trees into JSONL.
• references/chrome_same_origin.md: Chrome collection loop, checkpointing, and manifest contract.
• references/schema.md: normalized row and audit requirements.