clipsmith-x
MANDATORY — load
references/plan.mdbefore any browser or extraction action begins.
Use and Maintain the Existing Downloader
Start with the existing downloader. If it fails, diagnose and repair that implementation rather than abandoning the capture or building a competing downloader.
cd <clipsmith-repo>/skills/clipsmith-x
npx tsx scripts/run.ts \
--post-url "<url>" \
--bundle-dir "/tmp/x-<tweet_id>"
The sections below (image extraction, anti-detection, MHTML generation) are implementation documentation for the script itself, not instructions for you to re-implement. If the script is missing or cannot run, inspect the skill/source installation and dependencies first. A recoverable local error is a reason to repair the existing workflow, not to stop. Do not replace it with an untested standalone downloader.
When writing to a KB inbox, use <platform>-<stable-id> for both the directory
and capture.json.id (for X: x-<tweet_id>). Prefer an explicit --bundle-dir
to choose the destination; it produces a normalized bundle in one run. An
existing directory requires --overwrite; never overwrite a previous capture
just to retry a failed extraction.
Readable Public Pages and Extraction Failures
X serves both legacy data-testid="tweet" markup and modern article.group/post
markup. Login/signup buttons beside a readable public post are not a login wall.
The downloader supports both layouts and scopes modern content by the target
post permalink, excluding replies and nested quoted-post media. Quoted source
links are kept separately from the post body; unavailable engagement counts are
omitted rather than guessed as zero.
A downloader error is not the end of the user's task. On failure, load
references/recovery.md and continue autonomously through diagnosis, a justified
repair, verification, and the original capture/authorized sink. Do not wait for
the user to say “it opens for me” or ask them to confirm a routine local fix.
This applies to unfamiliar future layouts and runtime errors, not just the
specific selectors documented above. Preserve partial assets throughout.
Clipsmith Bundle Normalization
The copied downloader produces a raw post folder with post.md, images,
optional video, and optional MHTML. Before finalizing a Clipsmith capture job,
convert that raw folder into a bundle with the shared normalizer:
cd <clipsmith-repo>
uv run clipsmith normalize raw x "<raw_dir>" "<bundle_dir>" \
--source-url "<original_url>" \
--canonical-url "<canonical_url>" \
--title "<title>" \
--author "<author>" \
--captured-at "<iso8601_time>" \
--json
uv run clipsmith validate-bundle "<bundle_dir>" --json
The normalizer keeps post.md, creates or copies summary.md, preserves
ocr.md/ocr.txt as kind: "ocr-text" if present, moves image and video
files (.webp, .jpg, .jpeg, .png, .gif, .mp4, .mov, .webm) into
assets/, and writes capture.json with matching assets[] entries
(kind: "image" or kind: "video"). MHTML files are not moved into the
bundle — they are out of scope. The normalizer accepts raw_dir == bundle_dir
(in-place finalize); when the downloader wrote directly to the bundle path with
--flat/equivalent, pass the same path for both and add --overwrite.
Do not call uv run clipsmith capture finalize until capture.json exists and
validation succeeds.
Quality Evaluation
Use the committed eval profile and fixture before changing prompt, extraction, media, MHTML, post-type detection, t.co expansion, or normalization behavior:
cd <clipsmith-repo>/skills/clipsmith-x
node scripts/eval.mjs \
--fixture x-kingson-skill-runtime-text \
--profile x-kingson-skill-runtime-text
When live X access is available, also validate the known media and article profiles:
node scripts/eval.mjs \
--post_dir "/path/to/x/media-output-folder" \
--profile x-droidbuilds-single-image
node scripts/eval.mjs \
--post_dir "/path/to/x/article-output-folder" \
--profile x-geekcatx-chatgpt-codex-article
The committed fixture is a user-owned text-only post. External media and article
profiles are live eval profiles only; do not commit third-party full text,
images, video, or MHTML unless permission is explicit. Use
prompts/evaluate-capture.md for agent AI eval and compare against the fixture
baseline evals/ai-evals/x-kingson-skill-runtime-text.md when working in the
source repo. Packaged skill installs may omit eval fixtures and baselines.
URL Expansion (t.co Shortlinks)
Tweet links render as t.co shortlinks (e.g., https://t.co/yr4YXZ6SgU) — both in href attributes and textContent. post.md must output the full resolved URL (e.g., https://github.com/anthropics/skills/tree/main/skills/pptx).
Because Twitter's Content Security Policy (CSP) blocks cross-origin fetch() inside page.evaluate(), URL expansion must be done in Node.js:
- Collect all
<a>elementhrefattributes insidepage.evaluate() - Call
fetch(href, { redirect: "follow" })onhttps://t.co/*links from Node.js to get the final URL - Replace each anchor's
textContentwith the resolved URL, then extract full text
This step is integrated into extractTweetSnapshot() and must not be skipped.
Required Constraints
- Use browser automation only.
- Do not use X private APIs.
- Reuse manual-login session via unified Chrome CDP startup:
open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-labali". - Inherit the system proxy by default. Do not add
--no-proxy-serverunless direct access was explicitly selected. When normal Chrome works but capture media times out, compare startup flags and proxy routing before repeating downloads. - Prefer semantic extraction from visible page state and loaded resources.
- Download only target post assets: images plus optional post video.
- Generate
post.mdfor extracted text metadata. - Do not generate
manifest.json. - Preserve all query parameters for page navigation.
Anti-Detection Principles
X applies behavioral analysis to detect automation. Violations of these principles may result in account restrictions.
Core test — apply before every browser action:
"Would a real user do this, from this state, at this moment?" If no → skip it or slow it down. Re-navigating an already-open post, batch-extracting DOM nodes, issuing fresh HTTP requests for images the browser just loaded, using fixed delays — all fail this test.
Navigation:
- If the tab is already on the target post URL, skip
page.goto()entirely. - All fixed
waitForTimeoutvalues must be randomized (e.g.,base + Math.random() * range). - After navigating to a post, scroll down briefly to simulate reading, then scroll back up before interacting with images.
Image acquisition:
- Never issue new HTTP requests for images — the browser has already downloaded them.
- Register
page.on("response")BEFORE callingpage.goto()— images load during navigation. - Primary image source: DOM extraction from
[data-testid="tweetPhoto"]— handles both<img>elements (regular tweets) and CSSbackground-imagedivs (X Notes). - Fetch each URL via
fetch(url, {cache: 'force-cache'})inpage.evaluate(). - After download: deduplicate by SHA-256 content hash; remove files < 40% of median size.
MHTML Generation (CDP Page.captureSnapshot):
- Calls Chrome's built-in
Page.captureSnapshotcommand via Playwright CDPSession to generate standard MHTML - Before archiving, automatically removes UI elements unrelated to post content (login buttons, nav bars, recommended content, etc.)
cleanupPageForArchiveremoves: LoginForm, signup links, banner nav, sidebarColumn, app-bar, etc.- Text-only tweets (no images/video) skip MHTML and only generate
post.md - X Notes generate
article.mhtml+post.md; regular media posts save media files andpost.mdwithout MHTML.
Video:
- Before downloading, simulate user engagement: bring the tab to front, click the video element, wait 3–5 seconds (randomized) for buffering.
- Video download uses
page.request.get().
General:
- Always operate within the user's authenticated Chrome session (CDP reuse) — never launch a headless or separate browser.
- Never manipulate the DOM beyond what a user's own browser JS would do.
NEVER
- Never launch a new Chrome if CDP is already responding on port 9222.
- Never hijack a non-X browser tab — reuse X tab or open a new X tab.
- Never strip query params from post URL before navigating.
- Never use fixed (non-randomized) delays.
- Never retry after CAPTCHA or rate-limit signal.
- Never report success without verifying output files exist.
Success Criteria
A run is successful only when all conditions hold:
- An output folder is created at the explicit
--bundle-dir, or<output_dir>/<tweet_id>/for raw downloads. KB inbox bundles usex-<tweet_id>for both the directory andcapture.json.id. post.mdis generated in the folder with author, text, timestamp, URL.- Post image files are saved as
image_01.jpg,image_02.jpg, etc. - Post video file is saved as
video.mp4when the post contains video. article.mhtmlis generated for X Notes; regular media and text-only posts omit it.- Files are deduplicated by content hash.
- URL output and logs use canonical
x.comform.
Operational Mode
- Default: guided browser flow + semantic extraction + authenticated media download.
- Startup:
- Check if CDP responds:
curl -s http://localhost:9222/json/version - If CDP NOT responding → auto-launch Chrome immediately:
open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-labali"; wait 3s; verify CDP responds - If Chrome with remote debugging is already running, reuse it
- Find existing x.com tab → reuse it; if none → open new tab
- Check login state; if wall detected → guide user to complete login manually; continue in same session
- Check if CDP responds:
- Input:
- After startup, if
post_urlis missing → prompt user interactively - If
output_diris missing → prompt with default~/Downloads/x
- After startup, if
Resources
| When | Must load | Do NOT load |
|---|---|---|
| Always — at skill invocation start | references/plan.md |
references/architecture.md |
| Extraction returns wrong count or fails | references/recovery.md, then architecture if needed |
— |
| Video download unclear | references/architecture.md |
— |
See Resources table above for conditional loads.