Jd Ingest

JD Ingest (Stage 0 — LinkedIn Ingestion)

Inputs

Input validation & priority:

1. If file is provided (e.g. @linkedin-ingest-data.json) → use the JSON file as the data source. Skip HTML fetching entirely. The file should contain an array of job objects with fields: job_id, title, company, location, job_link, work_type, employment_type, salary, easy_apply, linkedin_matching, duplicate_check_key, comptitle, company_url, promoted.
2. If url is provided (no file) → fetch the listing page HTML via curl (guest/unauthenticated).
3. If neither is provided → ask the user.

Atlas browser workflow (recommended for logged-in data):

• Open the LinkedIn search page in Atlas browser (already logged in).
• Paste the prompt from ATLAS_BROWSER_PROMPT.md (located next to this file).
• Atlas outputs a JSON array + summary report.
• Save the JSON to a file in the project, then pass it here via file.

Execution

Step 1 — Load Stage 0 Framework

Fetch the Stage 0 framework from Notion:

• Stage 0 — LinkedIn ingest https://www.notion.so/Stage-0-LinkedIn-ingest-3358d41a5cff80d696dfcf3b9ac748d3

Read and follow the instructions in that page for the full extraction logic, duplicate handling, field mappings, and output format.

Step 2 — Acquire Job Data

Choose one path based on the input provided:

Path A — Atlas JSON file (preferred for logged-in data)

• Read the JSON file specified by file.
• Each element already contains job_id, title, company, location, job_link, duplicate_check_key, comptitle, linkedin_matching, work_type, etc.
• Proceed directly to Step 3 (duplicate check + create).

Path B — URL fetch (guest HTML, fallback)

• If url is provided → fetch the listing page HTML via curl.
• Parse job cards from the HTML (regex / HTML parsing).
• Extract fields: job_id, title, company, location, job_link, work_type, etc.
• Note: guest HTML may not contain all jobs visible when logged in, and will not contain LinkedIn match indicators.

Step 3 — Run the Stage 0 Pipeline

Execute the Stage 0 framework loaded in Step 1:

1. Read job data (from Atlas JSON or parsed HTML)
2. Normalize job fields
3. Generate Duplicate Check Keys (if not already present in JSON)
4. Check Job Applications DB for duplicates
5. Skip duplicates
6. Capture Linkedin-matching (from JSON field or HTML when available)
7. Create new records with Status = Saved

Step 3A — Strict Duplicate Validation (mandatory)

To avoid false duplicate reports, use this strict dedupe protocol for every job:

1. Search by exact job_id in the target data source.
2. If search returns no result, treat as candidate new.
3. If search returns one or more pages, fetch candidate page(s) and verify duplicate only when:
   • Job ID exactly equals the incoming job_id, or
   • Duplicate Check Key exactly equals incoming duplicate_check_key.
4. If neither exact match is present after fetch verification, treat as new (do not mark duplicate based on semantic similarity, title similarity, or company similarity).

Hard rule: never conclude "all duplicates" from search results alone; fetch-level exact validation is required.

Step 3B — Post-run Reconciliation (mandatory)

After create/skip decisions complete, run a verification pass before reporting:

1. For each input job_id, re-check existence in target data source.
2. Produce and verify these counts:
   • input_total
   • confirmed_existing_total
   • confirmed_created_this_run
   • confirmed_duplicates_skipped
   • confirmed_missing_total
3. Validation equation must hold:
   • confirmed_created_this_run + confirmed_duplicates_skipped == input_total
4. If equation fails, report run as incomplete/inconsistent (not successful), include missing IDs, and do not claim final duplicate/create totals.

Step 3C — Reliability and Timeout Handling

If Notion MCP calls time out or rate limit:

• Retry with smaller batches.
• Persist progress checkpoints (processed IDs, created IDs, skipped IDs).
• Resume from last checkpoint.
• Mark final status as partial if unresolved; never present partial outcomes as final.

Step 3D — Target Scope Confirmation

Before dedupe and reporting, explicitly confirm the target destination:

• Data source ID (e.g. collection://...)
• Database/view context if user references a specific view (e.g. "drop DB")

If ambiguous, ask once before mutating data.

Step 4 — Mandatory Ingest Log Prepend (top-of-page incremental update)

After each /JD-ingest run, you MUST prepend an incremental run summary to:

• https://www.notion.so/JD-ingest-log-3368d41a5cff80268c89c3bb4ab4056f

Rules:

1. Always prepend at top (newest entry first). Do not append.
2. Use notion-fetch first to get current page content and the exact old_str anchor.
3. Use notion-update-page with command: "update_content" to replace:
   • old_str = <current full content> (or a stable top prefix),
   • new_str = <new incremental block> + "\\n\\n" + <old content>.
4. Incremental block must include:
   • timestamp (YYYY-MM-DD HH:MM, local),
   • intake summary (scanned, duplicates, created),
   • grouped newly created links by Linkedin-matching (or No match).
5. If no new jobs were created, still prepend a short block stating created: 0.
6. If update times out/rate-limits:
   • retry in smaller payloads,
   • re-fetch before each retry to avoid stale old_str,
   • do not claim success unless fetch confirms the new top block exists.
7. Final response for /JD-ingest must explicitly state whether log prepend succeeded.

Expected Output

Follow the output format defined in the Stage 0 framework:

• Execution Summary — browser read, jobs extracted, duplicates checked, records created
• Intake Summary — total scanned, total duplicates skipped, total new records created
• New jobs grouped by Linkedin-matching — only newly created job links, grouped by match level
• Verification Block (required) — input_total, confirmed_created_this_run, confirmed_duplicates_skipped, confirmed_missing_total, and whether reconciliation equation passed