# Clipsmith X

> Download X (Twitter) post assets — text, images, video — to a local folder using browser automation with manual-login session reuse. Use when downloading an X post, saving post images, exporting post content, or archiving a tweet. Trigger phrases: "download x post", "save tweet", "x post images", "x video download", "twitter post assets".

- Skill: `octopusgarage/clipsmith-x` (Agent Skill, multi-file: 34 files)
- Install (CLI): `npx get-skill add octopusgarage/clipsmith-x`
- Raw SKILL.md: https://api.skillmd.com/api/skills/octopusgarage/clipsmith-x/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: OctopusGarage (https://skillmd.com/u/octopusgarage)
- Updated: 2026-11-18
- Page: https://skillmd.com/skills/octopusgarage/clipsmith-x

---


# clipsmith-x

> **MANDATORY — load `references/plan.md` before any browser or extraction action begins.**

## Use and Maintain the Existing Downloader

**Start with the existing downloader. If it fails, diagnose and repair that implementation rather than abandoning the capture or building a competing downloader.**

```bash
cd <clipsmith-repo>/skills/clipsmith-x
npx tsx scripts/run.ts \
  --post-url "<url>" \
  --bundle-dir "/tmp/x-<tweet_id>"
```

The sections below (image extraction, anti-detection, MHTML generation) are **implementation documentation for the script itself**, not instructions for you to re-implement. If the script is missing or cannot run, inspect the skill/source installation and dependencies first. A recoverable local error is a reason to repair the existing workflow, not to stop. Do not replace it with an untested standalone downloader.

When writing to a KB inbox, use `<platform>-<stable-id>` for both the directory
and `capture.json.id` (for X: `x-<tweet_id>`). Prefer an explicit `--bundle-dir`
to choose the destination; it produces a normalized bundle in one run. An
existing directory requires `--overwrite`; never overwrite a previous capture
just to retry a failed extraction.

## Readable Public Pages and Extraction Failures

X serves both legacy `data-testid="tweet"` markup and modern `article.group/post`
markup. Login/signup buttons beside a readable public post are not a login wall.
The downloader supports both layouts and scopes modern content by the target
post permalink, excluding replies and nested quoted-post media. Quoted source
links are kept separately from the post body; unavailable engagement counts are
omitted rather than guessed as zero.

A downloader error is not the end of the user's task. On failure, load
`references/recovery.md` and continue autonomously through diagnosis, a justified
repair, verification, and the original capture/authorized sink. Do not wait for
the user to say “it opens for me” or ask them to confirm a routine local fix.
This applies to unfamiliar future layouts and runtime errors, not just the
specific selectors documented above. Preserve partial assets throughout.

## Clipsmith Bundle Normalization

The copied downloader produces a raw post folder with `post.md`, images,
optional video, and optional MHTML. Before finalizing a Clipsmith capture job,
convert that raw folder into a bundle with the shared normalizer:

```bash
cd <clipsmith-repo>
uv run clipsmith normalize raw x "<raw_dir>" "<bundle_dir>" \
  --source-url "<original_url>" \
  --canonical-url "<canonical_url>" \
  --title "<title>" \
  --author "<author>" \
  --captured-at "<iso8601_time>" \
  --json
uv run clipsmith validate-bundle "<bundle_dir>" --json
```

The normalizer keeps `post.md`, creates or copies `summary.md`, preserves
`ocr.md`/`ocr.txt` as `kind: "ocr-text"` if present, moves image and video
files (`.webp`, `.jpg`, `.jpeg`, `.png`, `.gif`, `.mp4`, `.mov`, `.webm`) into
`assets/`, and writes `capture.json` with matching `assets[]` entries
(`kind: "image"` or `kind: "video"`). MHTML files are not moved into the
bundle — they are out of scope. The normalizer accepts `raw_dir == bundle_dir`
(in-place finalize); when the downloader wrote directly to the bundle path with
`--flat`/equivalent, pass the same path for both and add `--overwrite`.

Do not call `uv run clipsmith capture finalize` until `capture.json` exists and
validation succeeds.

## Quality Evaluation

Use the committed eval profile and fixture before changing prompt, extraction,
media, MHTML, post-type detection, t.co expansion, or normalization behavior:

```bash
cd <clipsmith-repo>/skills/clipsmith-x
node scripts/eval.mjs \
  --fixture x-kingson-skill-runtime-text \
  --profile x-kingson-skill-runtime-text
```

When live X access is available, also validate the known media and article
profiles:

```bash
node scripts/eval.mjs \
  --post_dir "/path/to/x/media-output-folder" \
  --profile x-droidbuilds-single-image

node scripts/eval.mjs \
  --post_dir "/path/to/x/article-output-folder" \
  --profile x-geekcatx-chatgpt-codex-article
```

The committed fixture is a user-owned text-only post. External media and article
profiles are live eval profiles only; do not commit third-party full text,
images, video, or MHTML unless permission is explicit. Use
`prompts/evaluate-capture.md` for agent AI eval and compare against the fixture
baseline `evals/ai-evals/x-kingson-skill-runtime-text.md` when working in the
source repo. Packaged skill installs may omit eval fixtures and baselines.

### URL Expansion (t.co Shortlinks)

Tweet links render as t.co shortlinks (e.g., `https://t.co/yr4YXZ6SgU`) — both in `href` attributes and `textContent`. **post.md must output the full resolved URL** (e.g., `https://github.com/anthropics/skills/tree/main/skills/pptx`).

Because Twitter's Content Security Policy (CSP) blocks cross-origin `fetch()` inside `page.evaluate()`, URL expansion must be done in Node.js:

1. Collect all `<a>` element `href` attributes inside `page.evaluate()`
2. Call `fetch(href, { redirect: "follow" })` on `https://t.co/*` links from Node.js to get the final URL
3. Replace each anchor's `textContent` with the resolved URL, then extract full text

This step is integrated into `extractTweetSnapshot()` and **must not be skipped**.

## Required Constraints

- Use browser automation only.
- Do not use X private APIs.
- Reuse manual-login session via unified Chrome CDP startup:
  `open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-labali"`.
- Inherit the system proxy by default. Do not add `--no-proxy-server` unless direct access was explicitly selected. When normal Chrome works but capture media times out, compare startup flags and proxy routing before repeating downloads.
- Prefer semantic extraction from visible page state and loaded resources.
- Download only target post assets: images plus optional post video.
- Generate `post.md` for extracted text metadata.
- Do not generate `manifest.json`.
- Preserve all query parameters for page navigation.

## Anti-Detection Principles

X applies behavioral analysis to detect automation. Violations of these principles may result in account restrictions.

**Core test — apply before every browser action:**
> "Would a real user do this, from this state, at this moment?"
> If no → skip it or slow it down. Re-navigating an already-open post, batch-extracting DOM nodes, issuing fresh HTTP requests for images the browser just loaded, using fixed delays — all fail this test.

**Navigation:**
- If the tab is already on the target post URL, skip `page.goto()` entirely.
- All fixed `waitForTimeout` values must be randomized (e.g., `base + Math.random() * range`).
- After navigating to a post, scroll down briefly to simulate reading, then scroll back up before interacting with images.

**Image acquisition:**
- Never issue new HTTP requests for images — the browser has already downloaded them.
- **Register `page.on("response")` BEFORE calling `page.goto()`** — images load during navigation.
- **Primary image source: DOM extraction from `[data-testid="tweetPhoto"]`** — handles both `<img>` elements (regular tweets) and CSS `background-image` divs (X Notes).
- Fetch each URL via `fetch(url, {cache: 'force-cache'})` in `page.evaluate()`.
- After download: deduplicate by SHA-256 content hash; remove files < 40% of median size.

**MHTML Generation (CDP Page.captureSnapshot):**
- Calls Chrome's built-in `Page.captureSnapshot` command via Playwright CDPSession to generate standard MHTML
- Before archiving, automatically removes UI elements unrelated to post content (login buttons, nav bars, recommended content, etc.)
- `cleanupPageForArchive` removes: LoginForm, signup links, banner nav, sidebarColumn, app-bar, etc.
- Text-only tweets (no images/video) skip MHTML and only generate `post.md`
- X Notes generate `article.mhtml` + `post.md`; regular media posts save media files and `post.md` without MHTML.

**Video:**
- Before downloading, simulate user engagement: bring the tab to front, click the video element, wait 3–5 seconds (randomized) for buffering.
- Video download uses `page.request.get()`.

**General:**
- Always operate within the user's authenticated Chrome session (CDP reuse) — never launch a headless or separate browser.
- Never manipulate the DOM beyond what a user's own browser JS would do.

## NEVER

- Never launch a new Chrome if CDP is already responding on port 9222.
- Never hijack a non-X browser tab — reuse X tab or open a new X tab.
- Never strip query params from post URL before navigating.
- Never use fixed (non-randomized) delays.
- Never retry after CAPTCHA or rate-limit signal.
- Never report success without verifying output files exist.

## Success Criteria

A run is successful only when all conditions hold:

1. An output folder is created at the explicit `--bundle-dir`, or `<output_dir>/<tweet_id>/` for raw downloads. KB inbox bundles use `x-<tweet_id>` for both the directory and `capture.json.id`.
2. `post.md` is generated in the folder with author, text, timestamp, URL.
3. Post image files are saved as `image_01.jpg`, `image_02.jpg`, etc.
4. Post video file is saved as `video.mp4` when the post contains video.
5. `article.mhtml` is generated for X Notes; regular media and text-only posts omit it.
6. Files are deduplicated by content hash.
7. URL output and logs use canonical `x.com` form.

## Operational Mode

- Default: guided browser flow + semantic extraction + authenticated media download.
- Startup:
  - Check if CDP responds: `curl -s http://localhost:9222/json/version`
  - If CDP NOT responding → auto-launch Chrome immediately: `open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-labali"`; wait 3s; verify CDP responds
  - If Chrome with remote debugging is already running, reuse it
  - Find existing x.com tab → reuse it; if none → open new tab
  - Check login state; if wall detected → guide user to complete login manually; continue in same session
- Input:
  - After startup, if `post_url` is missing → prompt user interactively
  - If `output_dir` is missing → prompt with default `~/Downloads/x`

## Resources

| When | Must load | Do NOT load |
|------|-----------|-------------|
| Always — at skill invocation start | `references/plan.md` | `references/architecture.md` |
| Extraction returns wrong count or fails | `references/recovery.md`, then architecture if needed | — |
| Video download unclear | `references/architecture.md` | — |

See Resources table above for conditional loads.

