Research Crawl X Posts

Research Crawl X Posts

Argument: $ARGUMENTS (required, see Runtime Inputs)

If $ARGUMENTS is empty or missing required fields, STOP and report which fields are absent.

Preamble

Runtime Inputs

Parse from $ARGUMENTS:

--handles=[h1,h2,...]            [JSON-style array of bare X/Twitter handles, non-empty; no leading @, no URL. A leading @ is stripped during normalization. Singular invocation expressed as a single-element array.]
--raw-output-dir={PATH}          [absolute path to a directory where per-handle rendered markdown files will be written]
--max-per-handle={N}             [positive integer per-handle cost-cap; MUST be ≥ 1, see §1.0 Step 3 + §6.0 row max-per-handle-zero]

# Optional power params (omit when not needed):
--days={N}                       [positive integer time window in days; default 1]

Validation rules (before §1.0):

• --handles, --raw-output-dir, and --max-per-handle mandatory. If any missing, STOP and report which.
• --handles must parse as a non-empty JSON array of strings. Else reject (handles-malformed).
• Normalize each handle by stripping a single leading @; the remainder must match the X handle shape (1-15 characters, [A-Za-z0-9_] only). First failing entry → STOP and report (handles-malformed).
• --raw-output-dir must be absolute. Reject relative paths (caller owns path resolution).
• --max-per-handle must parse as integer ≥ 1. Zero rejected per §6.0 max-per-handle-zero.
• --days must parse as a positive integer when supplied; default 1.

If any validation fails, STOP and report which field failed.

Global References

APIFY_ACTOR_ID    = apidojo/tweet-scraper           # chosen X/Twitter scraper actor (no-cookie, date-windowed search mode)
APIFY_CALL_ACTOR  = mcp__apify__call-actor           # Actor invocation, waitSecs 0-45 controls inline wait; no `async` flag exists
APIFY_GET_RUN     = mcp__apify__get-actor-run        # run-status polling endpoint (waitSecs blocks server-side until terminal)
APIFY_GET_ITEMS   = mcp__apify__get-dataset-items    # dataset retrieval, projects nested dot-notation fields via fields= (auto-flatten)
APIFY_ARTICLE_ACTOR = fastcrawler/x-twitter-article-to-markdown   # §3.0 Step 3 second-hop, recovers full X Article body that tweet-scraper gutted to a bare t.co (BRONZE PAY_PER_EVENT; tweetIds ≤10/run)

Leaf does not read tool guides at runtime, parameter choices in §1.0 Step 4 and the execution procedure in §3.0 already encode the pitfalls (cost-safety cap via per-handle maxItems, search-mode date targeting via from:/since:/until: since handle-mode ignores the date window, one Apify run per handle since maxItems is per-run-total, call-actor with waitSecs:45 inline + get-actor-run poll fallback (no async flag), {noResults} sentinel filtering, fetch-time narrow-field projection via the get-dataset-items fields= param, group by author.userName for the per-source split).

Article-enrichment pitfalls (§3.0 Step 3, APIFY_ARTICLE_ACTOR), all live-confirmed 2026-06-16: pass the HOST tweet id, the tweet's own id for a standalone article, quote.id for a quoted article, NEVER article.id or the /i/article/{id} id (both return empty md); the actor pads its dataset to a fixed 10-row minimum with mock filler ({id:0, text:"mock data"}, no tweet_id, no md), so keep only rows carrying a tweet_id AND a non-empty md and discard the rest; cost is a flat per-call floor (output always pads to 10, input caps at 10 → ≤$0.05/call at BRONZE regardless of real-id count), so batch all candidate ids for the whole invocation into one ≤10-id call (chunk into successive ≤10 calls only past 10 candidates); non-article ids return md:"" (clean negative, zero false positives); Apify is the only path, Exa/Firecrawl cannot reach the login-gated /i/article/ body.

Caller Isolation

Pure primitive, no agent spawn, no pipeline agents, no telemetry. The skill runs the Apify calls + dataset render directly in whatever context invokes it; the caller absorbs the MCP dataset context cost (raw tweet items land in the caller's window). Callers crawling many handles, or composing this leaf alongside other work, wrap the invocation in their own isolation:

Agent({subagent_type: "general-purpose", model: "sonnet", prompt: "Invoke /research-crawl-x-posts with <args>"})

Consumer skills document their own isolation choice. The run-pulse composer wraps each source-type leaf in one classifier sub-agent per source-type.

1.0 Context Capture

Step 1: Verify raw output directory parent exists

Check the parent directory of --raw-output-dir exists. If not, route to §6.0 (raw-output-parent-missing). Do not create parents, caller owns path resolution. The --raw-output-dir itself MAY or MAY NOT exist; §3.0 Step 1 creates it via mkdir -p before writing the first per-handle file.

Step 2: Normalize and validate handles

For each entry in --handles: strip a single leading @, then confirm the remainder matches the X handle shape (1-15 characters, [A-Za-z0-9_] only). Build handles[] of normalized bare handles preserving caller order. First entry that fails the shape → route to §6.0 (handles-malformed) with the offending value named.

Step 3: Validate --max-per-handle lower bound

Confirm --max-per-handle ≥ 1. If --max-per-handle == 0, route to §6.0 (max-per-handle-zero): pass a positive integer. --max-per-handle is the per-handle maxItems cap. Total estimated cost ≈ N_handles × max-per-handle × $0.0004 at the BRONZE/paid Apify tier; the FREE tier is 100× ($0.04/tweet) and only safe on a paid plan.

Step 4: Compute date window and construct per-handle Actor inputs

Compute the window:

since = (today - days)  formatted YYYY-MM-DD     # lower bound, inclusive
until = (today + 1 day)  formatted YYYY-MM-DD     # upper bound, EXCLUSIVE, +1 day so today is included

For each handle h in handles[], build a per-handle Actor input (one run per handle, maxItems is per-run-total, so a shared batched run starves trailing handles):

ACTOR_INPUT[h] = {
  searchTerms: ["from:{h} since:{since} until:{until}"],
  maxItems: {--max-per-handle},
  sort: "Latest",
  includeSearchTerms: true,
}

Cost-safety note: maxItems is the only verified-functional cost cap on this Actor. Do NOT add maxTotalChargeUsd (untested / non-functional on the sibling Actor; caps silently ignored). Do NOT set customMapFunction, the Actor README warns that using it for filtering triggers an automatic ban; filtering happens downstream, not in the input. Targeting MUST use searchTerms with from:/since:/until:, the twitterHandles + start/end path silently ignores the date window.

2.0 Plan & Confirm

Skill runs autonomously, no user pause. Pure primitive; executes directly in the caller's context. Composed leaves never gate parallelization; direct invocations log the execution config and proceed.

State the planned execution config in one structured log statement, then proceed directly to §3.0:

## /research-crawl-x-posts: executing crawl

handles:               {handles[]}                      # N normalized handles in caller-supplied order
raw_output_dir:        {--raw-output-dir}
max_per_handle:        {--max-per-handle}                # per-handle maxItems cap
days:                  {--days}                          # default 1
window:                {since} .. {until}                # until exclusive
actor:                 {APIFY_ACTOR_ID}
article_enrichment:    {APIFY_ARTICLE_ACTOR}             # §3.0 Step 3 second-hop; one batched ≤10-id call across all handles, fires only when article candidates exist
execution:             direct (pure primitive, no sub-agent spawn); one Apify run per handle
invocation:            call-actor waitSecs:45 inline → get-actor-run poll if not yet terminal (no `async` param)

3.0 Crawl & Enrich

Execute the crawl directly, no sub-agent spawn. Run the Apify apidojo/tweet-scraper Actor once per handle (search-mode, date-windowed), enrich any X Article posts with their full body (Step 3), then write per-handle rendered markdown files into --raw-output-dir. This step writes raw bytes only, it does NOT classify (Authored/Reposted/Mentioned), summarize, template-fill, or filter beyond the {noResults} sentinel and the per-handle cap; the composing router or domain orchestrator handles all downstream transformation. Article enrichment is raw-recovery, not transformation, it restores the verbatim article body that apidojo/tweet-scraper collapsed to a bare t.co link; it never classifies or summarizes.

Inputs (from §1.0)

Handles:               {handles[]}                       # N normalized handles in caller-supplied order
Raw output dir:        {--raw-output-dir}
Max per handle:        {--max-per-handle}                 # per-handle maxItems cap
Window:                {since} .. {until}                 # until exclusive
Actor:                 apidojo/tweet-scraper
Actor input (JSON):    {ACTOR_INPUT[h]}                   # one per handle, searchTerms from:{h} since:{since} until:{until}
invocation:            call-actor waitSecs:45 → poll get-actor-run if not terminal

The Actor inputs were auto-constructed in §1.0 Step 4. Do NOT substitute different params. Do NOT add maxTotalChargeUsd. Invoke via call-actor with waitSecs (0-45), there is NO async flag on the MCP tool. waitSecs:45 waits inline (this Actor finishes in ~5s, so the run is usually terminal on return); fall back to get-actor-run polling only when a run is still running.

Procedure

Execute these steps in order:

1. Ensure {--raw-output-dir} exists. Use Bash `mkdir -p {--raw-output-dir}`.

2. For EACH handle h in handles[] (one Apify run per handle):

   a. Call mcp__apify__call-actor with { actor: apidojo/tweet-scraper,
      input: ACTOR_INPUT[h], waitSecs: 45 }. The MCP tool has NO `async`
      flag, `waitSecs` (0-45) sets the inline wait. Capture `runId`, the
      dataset id at `storages.datasets.default.id`, and `status`. If
      `status === "SUCCEEDED"` already (typical, Actor finishes in ~5s),
      skip step 2b and go straight to step 2c.

   b. If step 2a's run was not yet terminal, poll
      mcp__apify__get-actor-run({ runId, waitSecs: 30 }), the param blocks
      server-side up to 30s per call, re-calling until `status` reaches one of:
      - `SUCCEEDED`, proceed to step 2c.
      - `FAILED` / `ABORTED` / `TIMED-OUT`, this handle's run failed. Skip
        steps 2c-2e; write an empty placeholder file for h (so callers can
        rely on one file per requested handle), set this handle's
        crawl_status: failed with diagnostic = run failure reason, continue
        to the next handle.
      Maximum poll duration per handle: 600 seconds. If still not terminal
      after 600s, abandon polling for h, treat as failed, write placeholder,
      continue with diagnostic = "Actor run exceeded 600s poll budget".

   c. Fetch this run's dataset via mcp__apify__get-dataset-items({
        datasetId: {storages.datasets.default.id}, limit: 1000,
        fields: "id,fullText,text,url,twitterUrl,createdAt,author.userName,isRetweet,isQuote,isReply,isPinned,inReplyToUsername,likeCount,retweetCount,replyCount,quoteCount,viewCount,quote.author.userName,quote.text,quote.id,article.id,article.title,article.previewText,retweet.author.userName,retweet.text,noResults,entities.media.media_url_https,extendedEntities.media.media_url_https,entities.media.type" }).
      - The `fields=` param projects AT FETCH TIME and supports nested
        dot-notation (server auto-flattens parent prefixes), so the raw
        ~198-field payload never enters caller context. This is the
        dataset-read tool's `fields` param, NOT the Actor INPUT `fields[]`
        (which is top-level-only; see step 2e note).
      - `quote.id` + the top-level `article.*` fields are the article-detection
        signals consumed by Step 3 (see step 2e); the rest carry the tweet body
        and structural flags.
      - Paginate via `offset` only when the dataset exceeds 1000 items
        (rare, bounded by max-per-handle).

   d. Filter out sentinel records where `noResults === true`. The remaining
      items are h's tweets. (A handle with no tweets in the window returns
      only the sentinel → tweets array is empty → crawl_status: empty.)

   e. The narrow field set is already projected by step 2c's `fields=` param
      (the raw record otherwise carries ~198 fields). The kept fields:
        id, fullText (fallback `text`), url (fallback `twitterUrl`),
        createdAt, author.userName, isRetweet, isQuote, isReply, isPinned,
        quote.author.userName, quote.text, quote.id, article.id, article.title,
        article.previewText, retweet.author.userName, retweet.text,
        inReplyToUsername, and media URLs
        (entities.media + extendedEntities.media `media_url_https`).
      Note, nested objects use `.text`, not `.fullText`: the nested quote/
      retweet payloads carry `quote.text` / `retweet.text`; `.fullText`
      exists only at the top level (verified via live probe). The
      get-dataset-items `fields=` param resolves all these nested keys; the
      top-level-only limitation belongs to the Apify Actor INPUT `fields[]`,
      not this dataset-read tool.
      Article-detection fields (live-confirmed 2026-06-16): a STANDALONE X
      Article populates a top-level `article` object (`article.id` /
      `article.title` / truncated `article.previewText`) while its `fullText`
      collapses to a bare t.co; a QUOTED article exposes NO `quote.article.*`
      object, only `quote.id` plus a link-collapsed `quote.text`. Both are
      consumed by Step 3.

3. Article-enrichment pass (cross-handle, single batched second-hop). After all
   handles are crawled and projected (Step 2), recover the full body of any X
   Article posts that `apidojo/tweet-scraper` collapsed to a bare t.co link.
   Detection + enrichment run ONCE across all handles' tweets, not per-handle,
   so the fixed cost floor (see Global References) is paid at most once per
   invocation.

   a. Detect article candidates across every projected tweet in this
      invocation. Two shapes, both live-confirmed:
      - STANDALONE authored article: top-level `article.id` present on the
        tweet. Candidate host id = the tweet's OWN `id`. (Free detection via the
        `article` object; the actor is still needed because `article.previewText`
        is truncated.)
      - QUOTED article: `isQuote === true` AND the quoted layer is
        link-collapsed. Mechanical test: strip every t.co/x.com URL and all
        whitespace from `quote.text`; the layer is link-collapsed when the
        residual is empty (no prose remains). Candidate host id = `quote.id`.
        (No `quote.article.*` object exists, so this is the only signal. A
        `quote.text` with residual prose is NOT a candidate, a missed real
        article is cheaper than burning the ≤10-id batch budget, and a false
        candidate that slips through returns `md:""` harmlessly.)
      Build a deduped set CANDIDATE_IDS, and a back-reference recording for each
      candidate id every tweet it belongs to plus whether each occurrence is
      that tweet's OWN body (standalone) or its QUOTED layer, needed for Step 4
      render injection. The same host id can occur in BOTH roles within one
      handle (a self-quote of one's own article); record every occurrence.

   b. If CANDIDATE_IDS is empty, skip enrichment entirely (no actor call, no
      cost); proceed to Step 4 with an empty ARTICLE_MD map.

   c. Enrich in batches of ≤10 ids (APIFY_ARTICLE_ACTOR caps tweetIds at 10).
      For each batch, call mcp__apify__call-actor with { actor:
      APIFY_ARTICLE_ACTOR, input: { tweetIds: [<batch>] }, waitSecs: 45 }. The
      actor takes ~30s; if not terminal on return, poll
      mcp__apify__get-actor-run({ runId, waitSecs: 30 }) up to a 120s budget per
      batch.

   d. Fetch each batch's dataset via mcp__apify__get-dataset-items({ datasetId,
      limit: 20, fields: "tweet_id,md" }). Build ARTICLE_MD as { host_id → md },
      keeping ONLY rows where `tweet_id` is present AND `md` is non-empty.
      Discard the fixed mock-padding rows ({id:0, text:"mock data"}, no
      tweet_id, no md) and empty-`md` rows (non-article ids, clean negative,
      zero false positives).

   e. An article-actor batch that FAILS / ABORTS / TIMES-OUT is NON-FATAL, the
      handle crawls already succeeded. Log a one-line diagnostic, leave those
      ids out of ARTICLE_MD, and let Step 4 fall back to the bare-link render
      for the affected tweets. Never fail the whole invocation on an enrichment
      miss. Set `article_enrichment_status: partial` in that case (else
      `clean`, or `skipped` when CANDIDATE_IDS was empty).

4. For each handle, render its projected tweet array to a markdown file inside
   {--raw-output-dir}. Filename rule:

     filename = `handle-{h}.md`     # h lowercased

   File structure per handle:

   ---
   type: X Posts Crawl Raw
   schema: ephemeral-raw-crawl
   source_tool: apidojo/tweet-scraper
   handle: {h}
   window_start: {since}
   window_end: {until}             # exclusive
   days: {--days}
   crawled_at: {ISO 8601 timestamp of this run}
   apify_run_id: {runId for this handle}
   tweet_count: {N tweets after sentinel filter}
   articles_enriched: {count of DISTINCT article bodies rendered in full for this handle, i.e. |rendered_article_ids| for this handle; back-references and a self-quote's second occurrence do NOT re-count; 0 when none}
   input_params:
     max_per_handle: {--max-per-handle}
   ---

   # X Posts, @{h}

   ## Tweet 1, @{author.userName}, {createdAt date}
   URL: {url}
   Engagement: {likeCount} likes · {retweetCount} reposts · {replyCount} replies · {quoteCount} quotes

   {tweet body, fullText; preserve original line breaks}

   ---

   ## Tweet 2, ...

   Rendering rules:
   - Each tweet is one H2 section separated by horizontal rules (`---`).
   - H2 heading: author handle + tweet date. Body is fullText verbatim
     (preserve newlines; keep bare t.co / x.com URLs as-is).
   - Render-once rule (article bodies): each recovered host id's full body is
     written at most ONCE per handle file, at the first occurrence (in tweet
     render order) where it is SUCCESSFULLY rendered in full. Track a
     `rendered_article_ids` set, adding a host id only when its full body is
     actually written (at render time, on success), NOT at detection time, so
     a first occurrence that hit an enrichment miss (bare-link fallback) never
     blocks a later occurrence from attempting its own full render. A later
     occurrence of an already-rendered host id (the self-quote case, a host id
     that is both a standalone tweet and another tweet's quoted layer) renders a
     one-line back-reference instead of duplicating the body: `Article '{title}'
, full body rendered at Tweet {N} above`, where `{title}` is `article.title`
     for a standalone host or the recovered body's first H1 heading for a quoted
     host (no `quote.article.title` field exists). The set is per-handle by
     design, it resets for each handle file so every file is self-sufficient;
     the same article quoted by two handles renders fully in both files.
   - STANDALONE X Article injection: when the tweet's own `id` is in ARTICLE_MD
     (Step 3, its `fullText` was a bare t.co) and not yet in
     `rendered_article_ids`, replace the body with a `Article, full body
     recovered:` marker line followed by the verbatim `ARTICLE_MD[id]` markdown
     (preserve its headings, links, and image refs), then add `id` to
     `rendered_article_ids`. If `id` is already in `rendered_article_ids`, emit
     the back-reference line per the render-once rule. When the tweet had an
     `article.id` candidate but enrichment missed (`id` not in ARTICLE_MD,
     actor failure/empty), fall back to the verbatim `fullText` (the bare t.co)
     and append ` (article body not recovered)`.
   - Engagement line is one line, dot-separated; omit a count when absent.
   - For a repost (`isRetweet: true`): H2 reflects the ORIGINAL author
     (@{retweet.author.userName}); insert
     `Reposted by: @{author.userName}` between the Engagement line and body;
     body is the original tweet text (retweet.text).
   - For a quote-tweet (`isQuote: true`): H2 reflects the watched handle as
     author; body is their commentary (fullText); the quoted original renders
     below in a blockquote under a `Quoted:` sub-heading showing
     @{quote.author.userName} + quote.text. QUOTED X Article injection: when
     `quote.id` is in ARTICLE_MD (Step 3, the quoted layer was a collapsed
     article) and not yet in `rendered_article_ids`, render the `Quoted:`
     sub-heading line showing @{quote.author.userName}, then the verbatim
     `ARTICLE_MD[quote.id]` markdown as its own block BELOW the sub-heading,
     NOT `>`-prefixed inside the blockquote. The recovered body carries its own
     `#` headings, `>` blockquotes, and `![image]` refs; wrapping it in an outer
     blockquote would mangle that nested markdown, so a quoted article breaks
     out of the blockquote exactly like a standalone one. Fence it between
     `<!-- quoted article: {title} -->` … `<!-- /quoted article -->` HTML
     comment markers so the boundary is unambiguous, then add `quote.id` to
     `rendered_article_ids`. If `quote.id` is already rendered, emit the
     back-reference line per the render-once rule (no second full body). On an
     enrichment miss, keep the bare-t.co `quote.text` in the normal `Quoted:`
     blockquote and append ` (article body not recovered)`.
   - For a reply (`isReply: true`): insert `Reply to: @{inReplyToUsername}`
     between the Engagement line and body.
   - For pinned tweets (`isPinned: true`): append ` (pinned)` to the H2.
   - For media-bearing tweets, render `Media: 1 image | 2 images | 1 video`
     as a one-line reference; do NOT embed media URLs (expiry unverified).
   - Classification (Authored / Reposted / Mentioned / Drop) is NOT performed
     here, the leaf surfaces the structural facts (the isRetweet/isQuote/
     isReply flags via the rendering above, original authors named); the
     caller or classifier sub-agent classifies.

   When `tweet_count: 0` for a handle (no tweets in window, quiet account):
   body has frontmatter + H1 + single line `No tweets returned for this handle
   in the configured window.` Per-handle `crawl_status: empty`, an empty
   window is a clean operational outcome, not a failure.

   When this handle's run failed (step 2b abort path): placeholder carries the
   file-level frontmatter + H1 + single line `Crawl failed for this handle,
   see diagnostic in return summary.` Per-handle `crawl_status: failed`.

5. Build `per_source_outputs[]` array, one entry per handle in caller order:

     per_source_outputs[i] = {
       handle:            {handles[i]},
       raw_output_path:   {--raw-output-dir}/handle-{handles[i]}.md,
       tweet_count:       {len(projected tweets for handles[i])},
       articles_enriched: {count of DISTINCT article bodies rendered in full for this handle = |rendered_article_ids| for this handle; back-references don't re-count},
       char_count:        {wc -c on the rendered file},
       crawl_status:      {"clean" | "empty" | "failed"},   # per-handle
       apify_run_id:      {runId for this handle},
       diagnostic:        {one-line note when crawl_status == failed; otherwise omit},
     }

   Status semantics:
   - `clean`, handle returned ≥1 tweet; run SUCCEEDED.
   - `empty`, handle returned zero tweets in window (only the sentinel).
   - `failed`, this handle's run FAILED/ABORTED/TIMED-OUT; placeholder written.

6. Compute aggregate fields:

     total_char_count        = sum(entry.char_count for entry in per_source_outputs)
     total_tweet_count       = sum(entry.tweet_count for entry in per_source_outputs)
     total_articles_enriched = sum(entry.articles_enriched for entry in per_source_outputs)
     overall_crawl_status = (
       "clean"   if all entries have crawl_status in {clean, empty}
       else "failed"  if all entries have crawl_status == "failed"
       else "partial"
     )

   Note: `empty` entries do NOT degrade overall status, an empty window is a
   clean outcome. Only mixed clean+failed or all-failed produces
   `partial`/`failed` aggregate. Article enrichment never affects
   `crawl_status`, an enrichment miss is logged via `article_enrichment_status`
   (Step 3e), not the crawl aggregate.

7. Read-back: list {--raw-output-dir} and confirm exactly N files exist (one per
   handle). Mismatch → set overall_crawl_status = "failed" with diagnostic =
   "post-write reconciliation failed: expected N files, found M".

Capture

After the read-back (step 7), capture the structured summary as CRAWL_RETURN and proceed to §4.0:

raw_output_dir:            {--raw-output-dir}
char_count:                <total across all per-handle files>     # aggregate
tweet_count:               <total across all per-handle groups>    # aggregate
articles_enriched:         <total across all handles, see step 6>
article_enrichment_status: <clean | partial | skipped>             # partial = ≥1 actor batch failed (Step 3e); skipped = no candidates
crawl_status:              <clean | partial | failed>              # aggregate
window_start:              {since}
window_end:                {until}                                 # exclusive
per_source_outputs:        [<one entry per handle, see step 5>]
diagnostic:                <one-line note when crawl_status != clean OR article_enrichment_status == partial; otherwise omit>

Do NOT surface raw tweet content or the Apify dataset to the caller, both are on disk at {--raw-output-dir} (rendered) or in Apify storage (raw JSON, 7-day retention). §5.0 returns the reference plus summary only.

4.0 Verify

Step 1: File-state verification

Verify the raw output directory and per-handle files:

• Directory exists at --raw-output-dir (use Bash: test -d {DIR}).
• One file per handle exists at each CRAWL_RETURN.per_source_outputs[i].raw_output_path.
• When entry.crawl_status == clean: file is non-empty (char_count > 200). File-level frontmatter alone exceeds 200 bytes, so any clean entry must have char_count > 200.
• When entry.crawl_status == empty: file is a placeholder with char_count > 0 (frontmatter + empty-window line).
• When entry.crawl_status == failed: file is present with char_count > 0 (placeholder body + diagnostic).

If any check fails (entry status clean but file empty/missing, file count ≠ N handles), route to §6.0 (agent-write-mismatch).

5.0 Summary

Return this structured summary to the caller (typically the /run-pulse composer):

raw_output_dir:            {CRAWL_RETURN.raw_output_dir}
char_count:                {CRAWL_RETURN.char_count}            # aggregate
tweet_count:               {CRAWL_RETURN.tweet_count}           # aggregate
articles_enriched:         {CRAWL_RETURN.articles_enriched}     # aggregate count of recovered article bodies
article_enrichment_status: {CRAWL_RETURN.article_enrichment_status}   # clean | partial | skipped
crawl_status:              {CRAWL_RETURN.crawl_status}          # clean | partial | failed
window_start:              {CRAWL_RETURN.window_start}
window_end:                {CRAWL_RETURN.window_end}            # exclusive
per_source_outputs:        {CRAWL_RETURN.per_source_outputs[]}
  # Each entry: { handle, raw_output_path, tweet_count, articles_enriched, char_count,
  #              crawl_status (clean|empty|failed per-handle), apify_run_id, diagnostic? }
diagnostic:                {CRAWL_RETURN.diagnostic if present}

Caller decides next action based on overall crawl_status (short-circuit on failed; proceed with extraction iteration on clean or partial) and per-handle crawl_status (skip extraction on entries with crawl_status == failed; treat empty as no-content-this-cycle). The leaf does not retry, the caller governs retry policy.

6.0 Error Handling