# Apify Scrapers > Social media and web scraping using Apify actors. Use this skill when scraping Twitter/X tweets, Reddit posts, LinkedIn posts, Instagram profiles/posts/reels, Facebook pages/posts/groups, TikTok videos, YouTube content, Google Maps businesses/reviews, contact enrichment (emails/phones from websites), or when auto-detecting URL type to scrape. Triggers on requests to scrape social media, get trending posts, extract business info, find contact details, or extract content from social URLs. Use when this capability is needed. - Skill: `tomevault-io/apify-scrapers` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/apify-scrapers` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/apify-scrapers/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/apify-scrapers --- # Apify Scrapers ## Overview Scrape content from major social platforms using Apify actors. Each platform has optimized settings for cost and quality. ## Quick Decision Tree ``` What do you want to scrape? │ ├── Social Media Posts │ ├── Twitter/X → references/twitter.md │ │ └── Script: scripts/scrape_twitter_ai_trends.py │ │ │ ├── Reddit → references/reddit.md │ │ └── Script: scripts/scrape_reddit_ai_tech.py │ │ │ ├── LinkedIn → references/linkedin.md │ │ └── Script: scripts/scrape_linkedin_posts.py │ │ │ ├── Instagram → references/instagram.md │ │ └── Script: scripts/scrape_instagram.py │ │ └── Modes: profile, posts, hashtag, reels, comments │ │ │ ├── Facebook → references/facebook.md │ │ └── Script: scripts/scrape_facebook.py │ │ └── Modes: page, posts, reviews, groups, marketplace │ │ │ ├── TikTok → references/multi-platform.md │ │ └── Script: scripts/scrape_multi_platform.py │ │ │ └── YouTube → references/multi-platform.md │ └── Script: scripts/scrape_multi_platform.py │ ├── Business/Places │ ├── Google Maps businesses → references/google-maps.md │ │ └── Script: scripts/scrape_google_maps.py │ │ └── Modes: search, place, reviews │ │ │ └── Contact info from websites → references/contact-enrichment.md │ └── Script: scripts/scrape_contact_info.py │ └── Extract: emails, phone numbers, social profiles │ ├── Auto-detect URL type → references/url-detect.md │ └── Script: scripts/scrape_content_by_url.py │ ├── Trend Analysis (NEW) │ └── Enriched trend analysis → workflows/trend-analysis.md │ └── Script: scripts/analyze_trends.py │ └── Features: velocity scoring, lifecycle staging, opportunity scoring │ └── Workflows (multi-step) ├── Lead generation → workflows/lead-generation.md ├── Influencer discovery → workflows/influencer-discovery.md ├── Competitor analysis → workflows/competitor-intel.md ├── Trend analysis → workflows/trend-analysis.md └── Competitor Ads Intelligence (NEW) → workflows/competitor-ads.md └── Script: scripts/scrape_competitor_ads.py └── Platforms: Facebook Ads Library, Google Ads Transparency └── Features: Spend estimates, creative analysis, benchmarking ``` ## Environment Setup ```bash # Required in .env APIFY_TOKEN=apify_api_xxxxx ``` Get your API key: https://console.apify.com/account/integrations ## Common Usage Patterns ### Scrape Twitter Trends ```bash python scripts/scrape_twitter_ai_trends.py --query "AI agents" --max-tweets 50 ``` ### Scrape Reddit Discussions ```bash python scripts/scrape_reddit_ai_tech.py --subreddits "MachineLearning,LocalLLaMA" --max-posts 100 ``` ### Scrape LinkedIn Author ```bash python scripts/scrape_linkedin_posts.py author "https://linkedin.com/in/username" --max-posts 30 ``` ### Auto-detect and Scrape URL ```bash python scripts/scrape_content_by_url.py "https://x.com/user/status/123456" ``` ### Scrape Instagram Profile ```bash python scripts/scrape_instagram.py profile "https://instagram.com/username" --max-posts 20 ``` ### Scrape Instagram Hashtag ```bash python scripts/scrape_instagram.py hashtag "#artificialintelligence" --max-posts 50 ``` ### Scrape Instagram Reels ```bash python scripts/scrape_instagram.py reels "https://instagram.com/username" --max-reels 30 ``` ### Scrape Facebook Page ```bash python scripts/scrape_facebook.py page "https://facebook.com/pagename" --max-posts 50 ``` ### Scrape Facebook Reviews ```bash python scripts/scrape_facebook.py reviews "https://facebook.com/pagename" --max-reviews 100 ``` ### Scrape Facebook Marketplace ```bash python scripts/scrape_facebook.py marketplace "laptops in san francisco" --max-items 30 ``` ### Scrape Google Maps Businesses ```bash python scripts/scrape_google_maps.py search "AI consulting firms in New York" --max-results 50 ``` ### Scrape Google Maps Reviews ```bash python scripts/scrape_google_maps.py reviews "ChIJN1t_tDeuEmsRUsoyG83frY4" --max-reviews 100 ``` ### Extract Contact Info from Websites ```bash python scripts/scrape_contact_info.py "https://example.com" --depth 2 ``` ### Bulk Contact Enrichment ```bash python scripts/scrape_contact_info.py --urls-file companies.txt --output contacts.json ``` ### Scrape Competitor Ads (Single Competitor) ```bash python scripts/scrape_competitor_ads.py "Nike" --platforms facebook google --country US --days 30 ``` ### Compare Multiple Competitors' Ads ```bash python scripts/scrape_competitor_ads.py "Nike" "Adidas" "Puma" --compare --output comparison.json ``` ### Discover Advertisers by Keyword ```bash python scripts/scrape_competitor_ads.py --search "running shoes" --country US --max-ads 200 ``` ### Filter Competitor Ads by Media Type ```bash python scripts/scrape_competitor_ads.py "Netflix" "Disney+" --platforms facebook --media-types video --days 7 ``` ### Analyze Trends (NEW) ```bash # Analyze specific topic with enrichments python scripts/analyze_trends.py "artificial intelligence" --sources google instagram tiktok --days 90 # Discover trending topics in category python scripts/analyze_trends.py --category technology --discover --top 50 # Compare multiple trends python scripts/analyze_trends.py "AI" "blockchain" "metaverse" --compare # Export HTML trend report python scripts/analyze_trends.py "sustainable fashion" --format html --output trend_report.html ``` ## Cost Estimates | Platform | Actor | Cost per Item | |----------|-------|---------------| | Twitter | kaitoeasyapi/twitter-x-data-tweet-scraper | ~$0.00025 | | Reddit | trudax/reddit-scraper | ~$0.001-0.005 | | LinkedIn | harvestapi/linkedin-post-search | ~$0.01-0.05 | | YouTube | streamers/youtube-scraper | ~$0.01-0.05 | | TikTok | clockworks/tiktok-scraper | ~$0.005 | | Instagram (profile) | apify/instagram-profile-scraper | ~$0.005 | | Instagram (posts) | apify/instagram-post-scraper | ~$0.002-0.005 | | Instagram (hashtag) | apify/instagram-hashtag-scraper | ~$0.002-0.005 | | Instagram (reels) | apify/instagram-reel-scraper | ~$0.005-0.01 | | Instagram (comments) | apify/instagram-comment-scraper | ~$0.001-0.003 | | Facebook (page) | apify/facebook-pages-scraper | ~$0.005-0.01 | | Facebook (posts) | apify/facebook-posts-scraper | ~$0.003-0.005 | | Facebook (reviews) | apify/facebook-reviews-scraper | ~$0.002-0.005 | | Facebook (groups) | apify/facebook-groups-scraper | ~$0.005-0.01 | | Facebook (marketplace) | apify/facebook-marketplace-scraper | ~$0.005-0.01 | | Google Maps (search) | compass/crawler-google-places | ~$0.01-0.02 | | Google Maps (place) | compass/google-maps-business-scraper | ~$0.01 | | Google Maps (reviews) | compass/google-maps-reviews-scraper | ~$0.003-0.005 | | Contact Enrichment | lukaskrivka/contact-info-scraper | ~$0.01-0.03 | | Google Trends | apify/google-trends-scraper | ~$0.01 | | Trend Analysis (multi) | Multiple actors | ~$0.50-1.50/run | | Facebook Ads Library | apify/facebook-ads-scraper | ~$0.75/1K ads | | Facebook Ads (alt) | curious_coder/facebook-ads-library-scraper | ~$0.50/1K ads | | Google Ads Transparency | lexis-solutions/google-ads-scraper | ~$1.00/1K ads | | Google Ads (alt) | xtech/google-ad-transparency-scraper | ~$0.80/1K ads | ## Output Location All scraped data saves to `.tmp/` with timestamped filenames: - `.tmp/twitter_ai_trends_YYYYMMDD.json` - `.tmp/reddit_ai_tech_YYYYMMDD.json` - `.tmp/linkedin_posts_YYYYMMDD_HHMMSS.json` ## Security Notes ### Credential Handling - Store `APIFY_TOKEN` in `.env` file (never commit to git) - Rotate API tokens periodically via Apify Console - Never log or print API tokens in script output - Use environment variables, not hardcoded values ### Data Privacy - Scraped data contains only publicly available content - Social media posts may include PII (names, handles, profile info) - Data is stored locally in `.tmp/` directory - No data is retained by Apify after actor run completes - Consider data minimization - only scrape what you need ### Access Scopes - Apify tokens have full account access (no granular scopes) - Use separate Apify accounts for different projects if needed - Monitor usage via Apify Console dashboard ### Compliance Considerations - **Terms of Service**: Respect each platform's ToS (Twitter, Reddit, LinkedIn) - **Rate Limiting**: Actors have built-in rate limiting to avoid bans - **Robots.txt**: Some actors may bypass robots.txt - use responsibly - **GDPR**: Scraped PII may be subject to GDPR if EU residents - **Ethical Use**: Only scrape public data; never bypass authentication - **Proxy Ethics**: Residential proxies should be used ethically ## Troubleshooting ### Common Issues #### Issue: Actor run failed **Symptoms:** Script terminates with "Actor run failed" or timeout error **Cause:** Invalid actor ID, insufficient proxy credits, or actor configuration issue **Solution:** - Verify the actor ID is correct in the script - Check Apify Console for actor run logs - Ensure proxy settings match actor requirements - Try running with default proxy settings first #### Issue: Empty results returned **Symptoms:** Script completes but returns 0 items **Cause:** Content blocked by platform, invalid query, or proxy being detected **Solution:** - Try a different proxy type (residential vs datacenter) - Simplify the search query - Reduce the number of results requested - Check if the platform is blocking scraping attempts #### Issue: Rate limited by platform **Symptoms:** Script fails with 429 errors or "rate limited" messages **Cause:** Too many requests in a short time period **Solution:** - Add delays between requests (actor settings) - Reduce concurrent requests - Use proxy rotation - Wait and retry after a cooldown period #### Issue: Invalid API token **Symptoms:** Authentication error or "invalid token" message **Cause:** Token expired, revoked, or incorrectly set **Solution:** - Regenerate API token in Apify Console - Verify token is correctly set in `.env` file - Check for leading/trailing whitespace in token - Ensure `APIFY_TOKEN` environment variable is loaded #### Issue: Proxy connection errors **Symptoms:** Connection timeout or proxy errors **Cause:** Proxy pool exhausted or geo-restriction issues **Solution:** - Switch proxy type (basic, residential, or datacenter) - Verify proxy credit balance in Apify Console - Try a different proxy country/region - Disable proxy to test if that's the root cause ## Resources ### Platform References - **references/twitter.md** - Twitter/X scraping details - **references/reddit.md** - Reddit scraping with subreddit targeting - **references/linkedin.md** - LinkedIn post scraping (author or search mode) - **references/instagram.md** - Instagram profile, posts, hashtag, reels, and comments scraping - **references/facebook.md** - Facebook page, posts, reviews, groups, and marketplace scraping - **references/multi-platform.md** - TikTok and YouTube scraping - **references/url-detect.md** - Auto-detect URL type and scrape ### Business/Places References - **references/google-maps.md** - Google Maps business search, place details, and reviews - **references/contact-enrichment.md** - Extract emails, phone numbers, and social profiles from websites ### Workflow References - **workflows/lead-generation.md** - Multi-step lead generation workflow - **workflows/influencer-discovery.md** - Find and analyze influencers across platforms - **workflows/competitor-intel.md** - Competitive intelligence gathering workflow - **workflows/trend-analysis.md** - Enriched multi-platform trend analysis with scoring ## Integration Patterns ### Scrape and Enrich **Skills:** apify-scrapers → parallel-research **Use case:** Scrape social media posts, then enrich with deep research **Flow:** 1. Scrape Twitter/Reddit for mentions of a topic 2. Extract company names or URLs from posts 3. Use parallel-research to get detailed info on each company ### Scrape and Summarize **Skills:** apify-scrapers → content-generation **Use case:** Create newsletter content from social media trends **Flow:** 1. Scrape trending AI posts from Twitter 2. Pass scraped data to content-generation summarize 3. Generate a formatted newsletter section ### Scrape and Archive **Skills:** apify-scrapers → google-workspace **Use case:** Save scraped data to Google Drive for team access **Flow:** 1. Scrape LinkedIn posts from target accounts 2. Format data as CSV or JSON 3. Upload to Google Drive client folder via google-workspace ### Trend Analysis + Content Strategy **Skills:** apify-scrapers (trend-analysis) → content-generation **Use case:** Identify trending topics and create content strategy **Flow:** 1. Run trend analysis: `python scripts/analyze_trends.py "AI productivity" --sources all` 2. Review lifecycle stage and opportunity score 3. Use content-generation to create content for high-opportunity trends 4. Focus on emerging trends with high velocity scores ### Competitive Trend Monitoring **Skills:** apify-scrapers (trend-analysis) → parallel-research **Use case:** Monitor competitor visibility in trending topics **Flow:** 1. Analyze industry trends: `python scripts/analyze_trends.py --category "your-industry" --discover` 2. Compare your brand vs competitors in those trends 3. Use parallel-research for deep dive on gaps 4. Generate competitive intelligence report --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/casper-studios) — claim your Tome and manage your conversions.