YouTube Domain TopN Videos
Overview
Use Patchright to crawl YouTube search results filtered to "Today", generate multi-query candidates from a chosen domain, rank by a recency-adjusted hot score, and export top-N items.
Run Workflow
- Install runtime dependencies.
- Update search queries when needed.
- Run the scraper.
- Read JSON/Markdown outputs and continue reporting.
Install
pip install patchright
python -m patchright install chromium
Configure Queries
Default query templates are in references/query_templates.txt.
- Use
{domain}placeholder in each line. - Example line:
{domain} tutorial - You can override with
--queries-fileto pass explicit queries.
Execute
python scripts/fetch_topn_domain_videos.py --domain "robotics" --top 10 --hours 24 --cookies-file H:/cookies/youtube.txt
Useful options:
--domain <text>: Required domain/topic, for examplecybersecurityor跨境电商.--queries-file <path>: Use explicit query lines directly.--query-templates-file <path>: Use{domain}templates.--cookies-file <path>: Load YouTube cookies (NetscapeorJSON).--max-results-per-query <n>: Increase candidate pool before ranking.--gl <country>and--hl <lang>: Control YouTube geo/language.--headed: Run non-headless for debugging.--output-json <path>and--output-md <path>: Change output files.
Default output layout:
- Project root directory:
outputs/YYYY/MM/DD/ - Filename pattern:
YYYYMMDD_top{N}_{domain}_{hours}h.jsonand.md
Output Contract
Each ranked record includes:
title: Video title.intro: Description snippet from search card.data: Channel, views, publish text, age-hours estimate, duration, hot score, domain, matched queries.link: Canonical YouTube watch link.
Troubleshooting
- If results are empty, increase
--scroll-roundsand--max-results-per-query. - If region bias is wrong, set
--glexplicitly (for exampleUS,JP,TW). - If YouTube UI changes, update DOM selectors in
scripts/fetch_topn_domain_videos.py.