BGGG X Data
Collect public X posts from the visible, rendered search timeline in the user's logged-in Chrome. Preserve one source package per query, then normalize locally.
VOC Project Layout（bggg 系列共用）
bggg VOC 系列 skill（bggg-data-amazon / bggg-data-reddit / bggg-data-x / bggg-voc-report / industry-orchestrator）共用一个项目文件夹，让多平台数据规整到同一处、下游分析零改路径。开工先确定项目根目录 <project>（用户指定，或新建 voc-<产品或主题slug>/），并从 <project> 根目录执行本 skill 的全部命令（下文相对路径都基于它）：
<project>/
PROJECT.md # 研究简报 + 决策日志（编排 skill 维护；单独使用可省）
config/ # 采集目标：amazon_targets.tsv / reddit_queries.tsv / x_queries.tsv / keywords.txt
work/<platform>/… # 各平台原始证据、attempt 日志、request plan、manifest
data/raw/ # 各平台规范化 JSONL（统一行契约，分析共用层）
data/clean|coded/ # 下游清洗与编码（industry-orchestrator 维护）
output/ # 报告与交付物（bggg-voc-report 写 output/report/）
本 skill 的落点：config/x_queries.tsv → work/x/（request plan、逐查询 source package）→ data/raw/x_multi_<date>.jsonl。
Workflow
- Prepare a tab-separated query file:
query lang round max_rows sort
sample-ingredient lang:en EN 1 250 latest
ボリュフィリン JP 1 200 latest
Build a deterministic plan:
python3 scripts/build_query_plan.py \
--queries config/x_queries.tsv \
--output work/x/request_plan.json
Use the Chrome plugin and follow its control skill. Select the user's Chrome explicitly, read its complete browser documentation, and reuse the browser binding. Never inspect or export cookies, local storage, profiles, passwords, or session stores.
Open the first planned search URL. Confirm from visible page state that X is signed in and the search timeline is available. If sign-in blocks the page, ask the user to sign in in Chrome; do not switch browser or bypass authentication.
For each query, collect only rendered cards from the visible DOM. Follow references/chrome_collection.md for the exact selectors, extraction function, scroll loop, checkpointing, and failure handling.
Save one unmodified query package immediately after each query:
work/x/source_json/001_EN.json
work/x/source_json/002_JP.json
Do not postpone all writes until the end of the run.
- Normalize and validate:
python3 scripts/normalize_x_dom.py \
--inputs work/x/source_json/*.json \
--output data/raw/x_multi_2026-07-26.jsonl \
--summary work/x/normalize_summary.json \
--keywords keywords.txt
- Report query hits, unique Tweet IDs, rows by language, missing-text/date/URL counts, duplicate observations, failure reasons, earliest/latest date, and collection limitations.
Required DOM Contract
Use these selectors only against rendered page content:
post card article[data-testid="tweet"]
post text [data-testid="tweetText"]
author block [data-testid="User-Name"]
canonical link time[datetime] inside a[href*="/status/"]
timestamp time[datetime]
engagement [role="group"][aria-label]
Validate the contract on the first query before scaling. If any required selector returns zero while the visible timeline contains posts, stop and inspect the current DOM rather than emitting empty success files.
Quality and Safety Rules
- Read visible DOM only. Do not intercept, call, or parse X's GraphQL/REST responses.
- Do not read, export, or persist browser cookies, tokens, local storage, credentials, or profile data.
- Preserve exact
text_raw, Tweet ID, canonical URL, timestamp, author handle, engagement label, query, language hint, and collection time.
- Treat X search as a visible sample, not a complete census. Record
latest versus top, query syntax, date bounds, caps, and stopping reason.
- Deduplicate by Tweet ID after preserving every query observation. Keep all matched queries and language hints in the normalized row.
- Separate consumers, promoters, sponsored UGC, media, and brands before calculating VOC prevalence.
- Prefer multiple narrow queries over one giant OR query. Split large historical searches by month or quarter.
- Use one tab and sequential queries by default. Avoid parallel browser tabs on the same account.
- Stop on challenge pages, suspicious-login prompts, rate limits, or repeated blank timelines. Record the failure and leave account recovery to the user.
- Never post, like, follow, reply, bookmark, or change account settings.
- Finalize every tab opened by the task.
Scale Guidance
- Default per-query cap: 250 posts.
- Scroll about 1,500 px, then wait 850–1,500 ms.
- Stop after 6 consecutive scrolls without a new Tweet ID.
- Also use a hard scroll cap, such as 180 iterations, to prevent runaway loops.
- For prevalence estimates, disclose X's search visibility limit and the query/date slicing scheme.
Degradation
If the logged-in Chrome session is unavailable or X blocks search, preserve the query plan and failure log. Do not substitute Tavily or search-engine snippets for original post text. Such tools may discover candidate URLs, but every quote must be revalidated against the original X page before entering the corpus.
Resources
scripts/build_query_plan.py: validate queries and build encoded X search URLs.scripts/normalize_x_dom.py: merge per-query packages, parse engagement, deduplicate, and emit normalized JSONL.references/chrome_collection.md: Chrome extraction loop and checkpoint contract.references/schema.md: source-package and normalized-row schemas.
1---2name: bggg-data-x3description: Collect auditable public X/Twitter posts by controlling the user's already logged-in Chrome, searching X's rendered web interface, scrolling visible results, and extracting original post text and metadata from DOM into normalized JSONL. Use for VOC research, social listening, multilingual keyword discovery, competitor monitoring, or historical search when X requires the user's browser session and cookies, credentials, internal GraphQL, paid APIs, or search-engine snippets must not be exported or used. X data layer of the bggg VOC suite (shared project folder; orchestrated by industry-orchestrator, reported by bggg-voc-report).4---56# BGGG X Data78Collect public X posts from the visible, rendered search timeline in the user's logged-in Chrome. Preserve one source package per query, then normalize locally.910## VOC Project Layout（bggg 系列共用）1112bggg VOC 系列 skill（bggg-data-amazon / bggg-data-reddit / bggg-data-x / bggg-voc-report / industry-orchestrator）共用一个项目文件夹，让多平台数据规整到同一处、下游分析零改路径。开工先确定项目根目录 `<project>`（用户指定，或新建 `voc-<产品或主题slug>/`），并从 `<project>` 根目录执行本 skill 的全部命令（下文相对路径都基于它）：1314```text15<project>/16 PROJECT.md # 研究简报 + 决策日志（编排 skill 维护；单独使用可省）17 config/ # 采集目标：amazon_targets.tsv / reddit_queries.tsv / x_queries.tsv / keywords.txt18 work/<platform>/… # 各平台原始证据、attempt 日志、request plan、manifest19 data/raw/ # 各平台规范化 JSONL（统一行契约，分析共用层）20 data/clean|coded/ # 下游清洗与编码（industry-orchestrator 维护）21 output/ # 报告与交付物（bggg-voc-report 写 output/report/）22```2324本 skill 的落点：`config/x_queries.tsv` → `work/x/`（request plan、逐查询 source package）→ `data/raw/x_multi_<date>.jsonl`。2526## Workflow27281. Prepare a tab-separated query file:2930```text31query lang round max_rows sort32sample-ingredient lang:en EN 1 250 latest33ボリュフィリン JP 1 200 latest34```3536Build a deterministic plan:3738```bash39python3 scripts/build_query_plan.py \40 --queries config/x_queries.tsv \41 --output work/x/request_plan.json42```43442. Use the Chrome plugin and follow its control skill. Select the user's Chrome explicitly, read its complete browser documentation, and reuse the browser binding. Never inspect or export cookies, local storage, profiles, passwords, or session stores.45463. Open the first planned search URL. Confirm from visible page state that X is signed in and the search timeline is available. If sign-in blocks the page, ask the user to sign in in Chrome; do not switch browser or bypass authentication.47484. For each query, collect only rendered cards from the visible DOM. Follow [references/chrome_collection.md](references/chrome_collection.md) for the exact selectors, extraction function, scroll loop, checkpointing, and failure handling.49505. Save one unmodified query package immediately after each query:5152```text53work/x/source_json/001_EN.json54work/x/source_json/002_JP.json55```5657Do not postpone all writes until the end of the run.58596. Normalize and validate:6061```bash62python3 scripts/normalize_x_dom.py \63 --inputs work/x/source_json/*.json \64 --output data/raw/x_multi_2026-07-26.jsonl \65 --summary work/x/normalize_summary.json \66 --keywords keywords.txt67```68697. Report query hits, unique Tweet IDs, rows by language, missing-text/date/URL counts, duplicate observations, failure reasons, earliest/latest date, and collection limitations.7071## Required DOM Contract7273Use these selectors only against rendered page content:7475```text76post card article[data-testid="tweet"]77post text [data-testid="tweetText"]78author block [data-testid="User-Name"]79canonical link time[datetime] inside a[href*="/status/"]80timestamp time[datetime]81engagement [role="group"][aria-label]82```8384Validate the contract on the first query before scaling. If any required selector returns zero while the visible timeline contains posts, stop and inspect the current DOM rather than emitting empty success files.8586## Quality and Safety Rules8788- Read visible DOM only. Do not intercept, call, or parse X's GraphQL/REST responses.89- Do not read, export, or persist browser cookies, tokens, local storage, credentials, or profile data.90- Preserve exact `text_raw`, Tweet ID, canonical URL, timestamp, author handle, engagement label, query, language hint, and collection time.91- Treat X search as a visible sample, not a complete census. Record `latest` versus `top`, query syntax, date bounds, caps, and stopping reason.92- Deduplicate by Tweet ID after preserving every query observation. Keep all matched queries and language hints in the normalized row.93- Separate consumers, promoters, sponsored UGC, media, and brands before calculating VOC prevalence.94- Prefer multiple narrow queries over one giant OR query. Split large historical searches by month or quarter.95- Use one tab and sequential queries by default. Avoid parallel browser tabs on the same account.96- Stop on challenge pages, suspicious-login prompts, rate limits, or repeated blank timelines. Record the failure and leave account recovery to the user.97- Never post, like, follow, reply, bookmark, or change account settings.98- Finalize every tab opened by the task.99100## Scale Guidance101102- Default per-query cap: 250 posts.103- Scroll about 1,500 px, then wait 850–1,500 ms.104- Stop after 6 consecutive scrolls without a new Tweet ID.105- Also use a hard scroll cap, such as 180 iterations, to prevent runaway loops.106- For prevalence estimates, disclose X's search visibility limit and the query/date slicing scheme.107108## Degradation109110If the logged-in Chrome session is unavailable or X blocks search, preserve the query plan and failure log. Do not substitute Tavily or search-engine snippets for original post text. Such tools may discover candidate URLs, but every quote must be revalidated against the original X page before entering the corpus.111112## Resources113114- `scripts/build_query_plan.py`: validate queries and build encoded X search URLs.115- `scripts/normalize_x_dom.py`: merge per-query packages, parse engagement, deduplicate, and emit normalized JSONL.116- `references/chrome_collection.md`: Chrome extraction loop and checkpoint contract.117- `references/schema.md`: source-package and normalized-row schemas.118
binggandata