# Jd Ingest

> Extract jobs from a LinkedIn jobs page, deduplicate against Notion Job Applications DB, and insert new records as Saved. Use when the user mentions JD ingest, LinkedIn job search, job ingestion, Stage 0, or bulk importing jobs from LinkedIn.

- Skill: `rayliu66/jd-ingest` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add rayliu66/jd-ingest`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rayliu66/jd-ingest/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: Rayliu66 (https://skillmd.com/u/rayliu66)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/rayliu66/jd-ingest

---


# JD Ingest (Stage 0 — LinkedIn Ingestion)

## Inputs

| Input | Description |
|-------|-------------|
| `url` | LinkedIn jobs page URL to extract jobs from |
| `file` | Path to a JSON file exported from Atlas browser (or similar logged-in browser agent) containing pre-extracted job data |

**Input validation & priority:**

1. If `file` is provided (e.g. `@linkedin-ingest-data.json`) → **use the JSON file** as the data source. Skip HTML fetching entirely. The file should contain an array of job objects with fields: `job_id`, `title`, `company`, `location`, `job_link`, `work_type`, `employment_type`, `salary`, `easy_apply`, `linkedin_matching`, `duplicate_check_key`, `comptitle`, `company_url`, `promoted`.
2. If `url` is provided (no file) → fetch the listing page HTML via curl (guest/unauthenticated).
3. If neither is provided → ask the user.

**Atlas browser workflow (recommended for logged-in data):**
- Open the LinkedIn search page in Atlas browser (already logged in).
- Paste the prompt from `ATLAS_BROWSER_PROMPT.md` (located next to this file).
- Atlas outputs a JSON array + summary report.
- Save the JSON to a file in the project, then pass it here via `file`.

## Execution

### Step 1 — Load Stage 0 Framework

Fetch the Stage 0 framework from Notion:

- **Stage 0 — LinkedIn ingest**
  https://www.notion.so/Stage-0-LinkedIn-ingest-3358d41a5cff80d696dfcf3b9ac748d3

Read and follow the instructions in that page for the full extraction logic, duplicate handling, field mappings, and output format.

### Step 2 — Acquire Job Data

Choose one path based on the input provided:

**Path A — Atlas JSON file (preferred for logged-in data)**
- Read the JSON file specified by `file`.
- Each element already contains `job_id`, `title`, `company`, `location`, `job_link`, `duplicate_check_key`, `comptitle`, `linkedin_matching`, `work_type`, etc.
- Proceed directly to Step 3 (duplicate check + create).

**Path B — URL fetch (guest HTML, fallback)**
- If `url` is provided → fetch the listing page HTML via curl.
- Parse job cards from the HTML (regex / HTML parsing).
- Extract fields: job_id, title, company, location, job_link, work_type, etc.
- Note: guest HTML may not contain all jobs visible when logged in, and will not contain LinkedIn match indicators.

### Step 3 — Run the Stage 0 Pipeline

Execute the Stage 0 framework loaded in Step 1:

1. Read job data (from Atlas JSON **or** parsed HTML)
2. Normalize job fields
3. Generate Duplicate Check Keys (if not already present in JSON)
4. Check Job Applications DB for duplicates
5. Skip duplicates
6. Capture `Linkedin-matching` (from JSON field or HTML when available)
7. Create new records with `Status = Saved`

### Step 3A — Strict Duplicate Validation (mandatory)

To avoid false duplicate reports, use this strict dedupe protocol for **every** job:

1. Search by exact `job_id` in the target data source.
2. If search returns no result, treat as **candidate new**.
3. If search returns one or more pages, fetch candidate page(s) and verify duplicate only when:
   - `Job ID` exactly equals the incoming `job_id`, **or**
   - `Duplicate Check Key` exactly equals incoming `duplicate_check_key`.
4. If neither exact match is present after fetch verification, treat as **new** (do not mark duplicate based on semantic similarity, title similarity, or company similarity).

**Hard rule:** never conclude "all duplicates" from search results alone; fetch-level exact validation is required.

### Step 3B — Post-run Reconciliation (mandatory)

After create/skip decisions complete, run a verification pass before reporting:

1. For each input `job_id`, re-check existence in target data source.
2. Produce and verify these counts:
   - `input_total`
   - `confirmed_existing_total`
   - `confirmed_created_this_run`
   - `confirmed_duplicates_skipped`
   - `confirmed_missing_total`
3. Validation equation must hold:
   - `confirmed_created_this_run + confirmed_duplicates_skipped == input_total`
4. If equation fails, report run as **incomplete/inconsistent** (not successful), include missing IDs, and do not claim final duplicate/create totals.

### Step 3C — Reliability and Timeout Handling

If Notion MCP calls time out or rate limit:

- Retry with smaller batches.
- Persist progress checkpoints (processed IDs, created IDs, skipped IDs).
- Resume from last checkpoint.
- Mark final status as partial if unresolved; never present partial outcomes as final.

### Step 3D — Target Scope Confirmation

Before dedupe and reporting, explicitly confirm the target destination:

- Data source ID (e.g. `collection://...`)
- Database/view context if user references a specific view (e.g. "drop DB")

If ambiguous, ask once before mutating data.

### Step 4 — Mandatory Ingest Log Prepend (top-of-page incremental update)

After each `/JD-ingest` run, you MUST prepend an incremental run summary to:

- `https://www.notion.so/JD-ingest-log-3368d41a5cff80268c89c3bb4ab4056f`

Rules:

1. **Always prepend at top** (newest entry first). Do not append.
2. Use `notion-fetch` first to get current page content and the exact `old_str` anchor.
3. Use `notion-update-page` with `command: "update_content"` to replace:
   - `old_str = <current full content>` (or a stable top prefix),
   - `new_str = <new incremental block> + "\\n\\n" + <old content>`.
4. Incremental block must include:
   - timestamp (`YYYY-MM-DD HH:MM`, local),
   - intake summary (`scanned`, `duplicates`, `created`),
   - grouped newly created links by `Linkedin-matching` (or `No match`).
5. If no new jobs were created, still prepend a short block stating `created: 0`.
6. If update times out/rate-limits:
   - retry in smaller payloads,
   - re-fetch before each retry to avoid stale `old_str`,
   - do not claim success unless fetch confirms the new top block exists.
7. Final response for `/JD-ingest` must explicitly state whether log prepend succeeded.

## Expected Output

Follow the output format defined in the Stage 0 framework:

- **Execution Summary** — browser read, jobs extracted, duplicates checked, records created
- **Intake Summary** — total scanned, total duplicates skipped, total new records created
- **New jobs grouped by Linkedin-matching** — only newly created job links, grouped by match level
- **Verification Block (required)** — `input_total`, `confirmed_created_this_run`, `confirmed_duplicates_skipped`, `confirmed_missing_total`, and whether reconciliation equation passed