Mianshiya Scraper

Mianshiya Scraper

Overview

Use this skill to turn a 面试鸭 question page such as https://www.mianshiya.com/bank/.../question/... into a local Markdown note plus local images. The bundled script uses Playwright so the user can complete QR-code login in a real browser window before content extraction.

Quick Start

Run:

python /Users/louis/.agents/skills/mianshiya-scraper/scripts/export_mianshiya.py \
  "https://www.mianshiya.com/bank/1991433644421414913/question/1991796895479013377" \
  --output-dir "/path/to/output"

Default behavior:

• Launch Chromium with a persistent profile at ~/.cache/mianshiya-scraper/profile.
• Let the user scan and finish login if no valid session exists.
• Save authenticated browser state under ~/.cache/mianshiya-scraper/storage-state.json.
• Extract the article/question content from the rendered DOM.
• Wait for answer/body Markdown content to appear before exporting.
• Download images into <output-dir>/assets/ by default.
• Write <output-dir>/<question title>.md with title, source URL, export timestamp, and Markdown content. Preserve the readable interview-question title, including Chinese text and numbering; sanitize only path-unsafe characters.

Workflow

1. Ask for or infer the target URL and output directory. Default the output directory to the current workspace when the user does not specify one.
2. Run scripts/export_mianshiya.py with the URL. Use --headed when the user likely needs to login; this is the default.
3. If the browser opens to a login or partially locked page, tell the user to scan the QR code or complete the site's login flow in that browser window. Continue after the script detects authenticated answer/body content.
4. Inspect the generated .md file briefly. Confirm the filename is based on the interview-question title and image links are relative to the shared assets/ folder.
5. If extraction is incomplete, rerun with --debug-html and inspect the saved HTML snapshot. Update the script's content selector heuristics rather than manually copying site content.

Script Notes

• Prefer the bundled script over ad hoc scraping. 面试鸭 has login and anti-bot checks, so authenticated browser rendering is more reliable than direct HTTP requests.
• Do not ask the user for passwords or tokens. Let them authenticate in the browser and reuse only the local Playwright profile/storage state.
• Use --force-login when the saved session is stale.
• Use --slug "custom-name" only when the user explicitly wants to override the title-based output filename.
• Use --assets-dir "custom-assets" only when the user explicitly wants a different image directory. The default should remain assets.
• Use --no-images only when the user explicitly wants text-only output.
• Use --allow-empty-answer only when the user explicitly accepts exporting a question shell without unlocked answer/body content.
• If Playwright browsers are missing and installation requires network access, request approval before running the install command.

Output Standard

Generate Markdown with:

• YAML frontmatter: title, source, exported_at, site.
• One # Title heading.
• Clean paragraphs, headings, lists, code blocks, blockquotes, tables where possible.
• Markdown filename should be the question title, e.g. 417. Java 中 Exception 和 Error 有什么区别？.md, not question-<id>.md.
• Images as relative Markdown links under the shared assets directory, e.g. ![alt](assets/image-abc123.png).
• No navigation, ads, login prompts, duplicated buttons, or unrelated sidebar content.

Resources

• scripts/export_mianshiya.py: browser login, extraction, image download, Markdown export.
• references/extraction.md: selector and troubleshooting notes for future maintenance.