Twitter Scraping
Auth Setup
All scrapers require an auth_token cookie from x.com (DevTools -> Application -> Cookies -> copy auth_token value).
Node.js Scraper API
npm install xactions -- all functions from src/scrapers/index.js.
import { createBrowser, createPage, loginWithCookie, scrapeProfile } from 'xactions';
const browser = await createBrowser();
const page = await createPage(browser);
await loginWithCookie(page, AUTH_TOKEN);
const profile = await scrapeProfile(page, 'nichxbt');
await browser.close();
| Function |
Purpose |
scrapeProfile(page, username) |
Profile data (bio, followers, following) |
scrapeFollowers(page, username, {limit}) |
Follower list |
scrapeFollowing(page, username, {limit}) |
Following list |
scrapeTweets(page, username, {limit}) |
User's tweets |
searchTweets(page, query, {limit}) |
Search results |
scrapeHashtag(page, tag, {limit}) |
Hashtag tweets |
scrapeThread(page, tweetUrl) |
Thread tweets |
scrapeMedia(page, username, {limit}) |
Media URLs |
exportToJSON(data, path) / exportToCSV(data, path) |
File export |
Browser Console Scripts
Standalone IIFEs -- paste into DevTools console on x.com. No dependencies.
| Script |
File |
Navigate to |
| Download video |
src/scrapers/videoDownloader.js |
Tweet with video |
| Export bookmarks |
src/scrapers/bookmarkExporter.js |
x.com/i/bookmarks |
| Unroll thread |
src/scrapers/threadUnroller.js |
Any thread |
| Find viral tweets |
src/scrapers/viralTweets.js |
User profile |
| Scrape followers |
scripts/scrapeFollowers.js |
x.com/USER/followers |
| Scrape following |
scripts/scrapeFollowing.js |
x.com/USER/following |
| Scrape profile |
scripts/scrapeProfile.js |
User profile |
| Scrape search |
scripts/scrapeSearch.js |
Search results |
| Scrape hashtag |
scripts/scrapeHashtag.js |
x.com/hashtag/TAG |
| Scrape likes |
scripts/scrapeLikes.js |
x.com/USER/likes |
| Scrape likers |
scripts/scrapeLikers.js |
Any tweet |
| Scrape replies |
scripts/scrapeReplies.js |
Any tweet |
| Scrape media |
scripts/scrapeMedia.js |
User media tab |
| Scrape DMs |
scripts/scrapeDMs.js |
x.com/messages |
| Scrape notifications |
scripts/scrapeNotifications.js |
x.com/notifications |
| Scrape list |
scripts/scrapeList.js |
Any X List |
| Scrape bookmarks |
scripts/scrapeBookmarks.js |
x.com/i/bookmarks |
| Export to CSV |
scripts/exportToCSV.js |
After any scrape |
| Link scraper |
scripts/linkScraper.js |
Any timeline |
| Quote retweets |
scripts/scrapeQuoteRetweets.js |
Any tweet |
Interactive browser toolbox (recommended for console scraping)
For paste-in-console scraping, prefer scripts/twitter/scraper-toolbox.js: an on-page
control panel (start/pause/stop, live filters, user exclusions, keyword rules) that
captures X's own GraphQL responses, so counts are exact integers and long posts are
untruncated. Exports JSON/CSV/Markdown/TXT/HTML and copies to clipboard. Works on any
timeline: profile, search, list, likes, bookmarks, home. Docs:
scripts/twitter/README-scraper-toolbox.md.
MCP Scraping Tools
All scrapers available via the MCP server for AI agents:
| Tool |
Description |
x_get_profile |
Scrape user profile |
x_get_followers |
Get follower list |
x_get_following |
Get following list |
x_get_tweets |
Get user tweets |
x_search_tweets |
Search tweets |
x_get_thread |
Unroll thread |
x_download_video |
Extract video URL |
Notes
- Always call
browser.close() when done with Node.js scrapers
- Browser scripts stop on page navigation -- stay on the page while running
- All scripts support JSON and CSV export
- Increase
limit parameter for deeper scraping (default varies by function)
- Rate limiting is built in -- avoid reducing delays below defaults
1---2name: twitter-scraping3description: Scrapes X/Twitter data without API access using Puppeteer stealth and browser console scripts. Extracts profiles, followers, following lists, tweets, search results, hashtags, threads, media, bookmarks, notifications, DMs, likes, and viral tweets. Exports to JSON/CSV. Use when collecting, exporting, or analyzing Twitter data.4license: Apache-2.05---67# Twitter Scraping89## Auth Setup1011All scrapers require an `auth_token` cookie from x.com (DevTools -> Application -> Cookies -> copy `auth_token` value).1213## Node.js Scraper API1415`npm install xactions` -- all functions from `src/scrapers/index.js`.1617```javascript18import { createBrowser, createPage, loginWithCookie, scrapeProfile } from 'xactions';1920const browser = await createBrowser();21const page = await createPage(browser);22await loginWithCookie(page, AUTH_TOKEN);2324const profile = await scrapeProfile(page, 'nichxbt');25await browser.close();26```2728| Function | Purpose |29|----------|---------|30| `scrapeProfile(page, username)` | Profile data (bio, followers, following) |31| `scrapeFollowers(page, username, {limit})` | Follower list |32| `scrapeFollowing(page, username, {limit})` | Following list |33| `scrapeTweets(page, username, {limit})` | User's tweets |34| `searchTweets(page, query, {limit})` | Search results |35| `scrapeHashtag(page, tag, {limit})` | Hashtag tweets |36| `scrapeThread(page, tweetUrl)` | Thread tweets |37| `scrapeMedia(page, username, {limit})` | Media URLs |38| `exportToJSON(data, path)` / `exportToCSV(data, path)` | File export |3940## Browser Console Scripts4142Standalone IIFEs -- paste into DevTools console on x.com. No dependencies.4344| Script | File | Navigate to |45|--------|------|-------------|46| Download video | `src/scrapers/videoDownloader.js` | Tweet with video |47| Export bookmarks | `src/scrapers/bookmarkExporter.js` | `x.com/i/bookmarks` |48| Unroll thread | `src/scrapers/threadUnroller.js` | Any thread |49| Find viral tweets | `src/scrapers/viralTweets.js` | User profile |50| Scrape followers | `scripts/scrapeFollowers.js` | `x.com/USER/followers` |51| Scrape following | `scripts/scrapeFollowing.js` | `x.com/USER/following` |52| Scrape profile | `scripts/scrapeProfile.js` | User profile |53| Scrape search | `scripts/scrapeSearch.js` | Search results |54| Scrape hashtag | `scripts/scrapeHashtag.js` | `x.com/hashtag/TAG` |55| Scrape likes | `scripts/scrapeLikes.js` | `x.com/USER/likes` |56| Scrape likers | `scripts/scrapeLikers.js` | Any tweet |57| Scrape replies | `scripts/scrapeReplies.js` | Any tweet |58| Scrape media | `scripts/scrapeMedia.js` | User media tab |59| Scrape DMs | `scripts/scrapeDMs.js` | `x.com/messages` |60| Scrape notifications | `scripts/scrapeNotifications.js` | `x.com/notifications` |61| Scrape list | `scripts/scrapeList.js` | Any X List |62| Scrape bookmarks | `scripts/scrapeBookmarks.js` | `x.com/i/bookmarks` |63| Export to CSV | `scripts/exportToCSV.js` | After any scrape |64| Link scraper | `scripts/linkScraper.js` | Any timeline |65| Quote retweets | `scripts/scrapeQuoteRetweets.js` | Any tweet |6667## Interactive browser toolbox (recommended for console scraping)6869For paste-in-console scraping, prefer `scripts/twitter/scraper-toolbox.js`: an on-page70control panel (start/pause/stop, live filters, user exclusions, keyword rules) that71captures X's own GraphQL responses, so counts are exact integers and long posts are72untruncated. Exports JSON/CSV/Markdown/TXT/HTML and copies to clipboard. Works on any73timeline: profile, search, list, likes, bookmarks, home. Docs:74`scripts/twitter/README-scraper-toolbox.md`.7576## MCP Scraping Tools7778All scrapers available via the MCP server for AI agents:7980| Tool | Description |81|------|-------------|82| `x_get_profile` | Scrape user profile |83| `x_get_followers` | Get follower list |84| `x_get_following` | Get following list |85| `x_get_tweets` | Get user tweets |86| `x_search_tweets` | Search tweets |87| `x_get_thread` | Unroll thread |88| `x_download_video` | Extract video URL |8990## Notes91- Always call `browser.close()` when done with Node.js scrapers92- Browser scripts stop on page navigation -- stay on the page while running93- All scripts support JSON and CSV export94- Increase `limit` parameter for deeper scraping (default varies by function)95- Rate limiting is built in -- avoid reducing delays below defaults
nirholas