Extract Advisorhub Articles

Extract AdvisorHub articles → Harper

This skill is the rich-extraction phase. It reads the saved WordPress posts, asks you (the Claude running this session) to read each article and produce a structured JSON document, then runs a deterministic Python loader that resolves entities against Harper and upserts.

ingest-advisorhub is the cheap regex-only path. This skill is its upgrade.

Pipeline

   Article body (prose)
        │
        │  (you read it; you produce JSON
        │   matching schema-guide.md)
        ▼
   research/extractions/<wpId>.json
        │
        │  scripts/load_extractions.py
        │  (resolver + upserts, idempotent)
        ▼
   Harper (Article, Advisor, Firm, Team, …)
        │
        ▼
   research/extractions/.loaded/<wpId>.json

Steps to follow when this skill is invoked

0. Pre-flight

Make sure Harper is up and the wpjson corpus exists:

npm run status

If Harper isn't running, ask the user whether to bootstrap. (npm run bootstrap will install/start it.)

If research/wpjson/ is empty, the user probably hasn't crawled yet — suggest the /ingest-advisorhub skill first to populate the corpus, then come back here.

1. Find pending articles

python3 scripts/extract_helper.py find-pending

That lists articles with a saved wpjson record but no extraction file yet. Output is human-readable; for scripting use --format tsv.

If nothing's pending, you're done — report that and stop.

2. Decide a batch size

You can comfortably handle about 20–30 articles per session before context gets tight (article body ≈ 2K tokens × N + the schema overhead). Pick a number; if there are more than ~30 pending, do a batch and tell the user to re-invoke for the rest.

3. For each article in the batch

For each pending wpId:

3a. Read the article body

python3 scripts/extract_helper.py show <wpId>

That prints title, URL, publication date, and body text. Read it.

3b. Extract entities into JSON

Read schema-guide.md once at the start of the batch — it defines the output document shape.

Read examples.md once — it contains worked examples you should pattern-match against.

For each article, write the extraction to:

research/extractions/<wpId>.json

Rules of the road:

• Every fact you put in fields must be supported by an exact phrase from the article body. Put that phrase in a field_assertions entry with the corresponding field, quote, and confidence.
• If you cannot find a quote for a fact, don't include the fact.
• Use confidence: "inferred" when you've derived a value (e.g., year of registration → industryStartDate); "derived" for computed values (e.g., yearsExperience from start date); "asserted" for verbatim values.
• For an opinion piece or editorial with no extractable entities, set article.has_extractable_content: false and emit empty arrays for every entity type (or just the article block + that flag).

3c. Quick self-check before moving on

Before writing the file, sanity-check:

• Every entity has a natural_key block.
• Every field_assertions entry's quote substring actually appears in the article body.
• Cross-references between entities use the natural-key fields (advisor_legal_name, firm_canonical_name, team_name, disclosure_local_key) — NOT UUIDs (those are assigned by the loader).

4. Load

Once all extraction files for the batch exist:

python3 scripts/load_extractions.py

Or for a single article:

python3 scripts/load_extractions.py --wpid <wpId>

The loader resolves IDs against Harper (matching existing advisors by name + employment overlap, existing firms by canonical name, existing disclosures by (advisor, type, date)), then upserts. On success it moves the extraction file to research/extractions/.loaded/.

5. Spot-check & report

After loading, run a couple of sanity queries:

npm run verify

…then report to the user (≤ 6 lines):

• How many articles were extracted in this batch.
• Resolver outcomes from the loader's stats line, e.g., advisor_matched=4 advisor_minted=2 firm_matched=8 firm_minted=1. A high *_matched ratio is good — it means the resolver is recognising existing entities instead of creating duplicates.
• Any articles that were skipped (no extractable content, or hit a validation issue).

How idempotency works at every layer

What this skill does NOT do

• Crawl new articles — that's /ingest-advisorhub. Run that first if there's nothing in research/wpjson/.
• Override existing data beyond an upsert — if seed.py loaded a Cairnes advisor and the resolver matches him, his row gets overwritten with the new fields, but his ID and dependent rows stay intact.
• Resolve fuzzy name matches automatically when there are multiple candidates. The resolver only auto-merges on (1) an exact CRD match, (2) a single name candidate, or (3) a name + firm-history overlap. Anything else mints a new ID; the user can manually merge later if needed.
• Process more than ~30 articles in one session. Tell the user to re-invoke for additional batches.