# Ocas Weave > Private provenance-backed social graph. Maintains queryable records of people, relationships, preferences, and shared experiences for recall, gifting, hosting, introductions, and serendipity. Use for storing or retrieving facts about a person, recording a relationship, or discovering connections between people. Not for sending messages (use Dispatch), calendar management (use Sands), OSINT research (use Scout), or web research without a social graph need (use Sift). - Skill: `indigokarasu/ocas-weave` (Agent Skill, multi-file: 7 files) - Install (CLI): `npx skillmds@latest add indigokarasu/ocas-weave` - Raw SKILL.md: https://api.skillmd.com/api/skills/indigokarasu/ocas-weave/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Research & Search - License: MIT - Author: indigokarasu (https://skillmd.com/u/indigokarasu) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/indigokarasu/ocas-weave --- # Weave Weave maintains a private, provenance-backed social graph of people, relationships, preferences, and shared experiences — queryable for meeting prep, gift ideas, hosting, introductions, city connections, and serendipity discovery. Every stored fact carries source type, reference, timestamp, and confidence score; the graph never silently merges two person records and never writes back to external systems without explicit per-sync approval. ## When to Use - Contact management and relationship tracking - Social graph queries (who knows whom, how) - Contact enrichment from multiple sources - Store or update information about a person, relationship, or preference - Prepare for a meeting, dinner, or introduction - Find connections in a given city - Generate gift ideas grounded in known preferences - Discover serendipity connections between people - Sync contacts from Google Contacts or Clay ## When NOT to Use - Sending messages or emails (use Dispatch) - Calendar management (use Sands) - OSINT research (use Scout) - Knowledge graph entity resolution - Web research without a social graph need — use Sift - CRM or sales pipeline automation - Personality profiling without evidence ## Auth Rule — Only **Weave exclusively uses 's Google auth (``).** Never use the agent's account for any Weave operation. The `TOKEN_PATH` in `google_sync.py` is hardcoded to `.json`. Violation silently fetches wrong contact data. **ALWAYS sync Contacts after changes** via `scripts/google_sync.py`. This is the canonical sync path — never skip it after a contact mutation. ## Responsibility Boundary Weave owns the social relationship graph: people, relationships, preferences, and shared experiences. It is the only skill that writes to its SQLite database (`weave.sqlite`). **Backend**: SQLite with WAL mode via `weave_sqlite.WeaveDB`. Replaced LadybugDB in June 2026. See `references/sqlite-backend-research.md` for the evaluation and migration details. Weave does not: perform OSINT research (Scout), manage calendars (Sands), or organize files (Bower). ## Ontology types - **Entity/Person** — people in the social graph. Weave extracts and manages Person entities exclusively. Weave may optionally include Chronicle signals in journal payloads for Person nodes with high-confidence identity markers. ## SQLite usage guide Read `references/schemas.md` for the Python usage pattern and schema details. Read `references/query_patterns.md` for all SQL query templates. Import: `from weave_sqlite import WeaveDB` ## Storage layout See `references/schemas.md` for the storage layout and record schemas. See `references/config-defaults.md` for default config. ## Database rules SQLite with WAL mode. Multiple concurrent readers + single writer. No lock contention — any process can read while another writes. If a write fails, surface the error, do not retry silently. ## Auto-initialization Every command that opens the database runs `_ensure_init()` first via `WeaveDB.__init__()`. No manual init command needed. ## Commands - **weave.upsert.person** — Add or update a person. Auto-inits DB on first call. - **weave.upsert.relationship** — Add or update a `Knows` edge. Confirm both Person nodes exist first. - **weave.upsert.preference** — Store a provenance-backed preference. - **weave.import.csv** — Bulk import contacts via `COPY FROM`. Read `references/import_export.md`. - **weave.query** — Query the graph. Modes: `lookup`, `connection`, `serendipity`, `city`, `summarize`, `gift`. Return only stored facts with provenance. - **weave.attach** — Query an external skill database read-only. - **weave.export** — Export data via `COPY TO`. - **weave.sync.google-contacts** — Bidirectional Google Contacts sync. Read `references/connectors.md`. - **weave.sync.clay** — Bidirectional sync with Clay. - **weave.project.vcard** — Generate vCard 4.0 draft. - **weave.writeback.contacts** — Push records to Google Contacts or Clay. Enabled by default; enrichment-derived values are withheld so only owner-sourced data is pushed. - **weave.init** — Diagnostic and repair. - **weave.status** — Report graph health and config state. - **weave.journal** — Write journal for the current run. - **weave.update** — Pull latest skill package from GitHub. ## Workflow The Weave social graph pipeline: **record → enrich → query → discover**. - [ ] Record people, relationships, preferences, and shared experiences - [ ] Enrich contacts via Scout/Sift/Sherlock pipeline - [ ] Query the graph for recall, gifting, hosting, introductions - [ ] Discover serendipitous connections between people ## Run completion After every Weave command: - [ ] Persist any new or updated records to the database. - [ ] Log material decisions to `decisions.jsonl`. - [ ] Write journal via `weave.journal`. - [ ] **Read-back verification**: After every write, immediately query the DB by primary key. Confirm written data matches intent. Never claim success unconfirmed. ## Provenance Every written fact requires: `source_type` (direct / inferred / imported / user-stated), `source_ref`, `record_time` (ISO 8601), `confidence` (0.0–1.0). ## Enrichment Pipeline Execution ### Pre-Enrichment Checklist 1. Run inbound Google sync 2. Check SearXNG health 2.5. **Probe ALL discovery sources before committing to the run**: `python3 scripts/discovery_probe.py`. If every source reports down (web_search empty, SearXNG engines suspended, DDG anomaly-blocked, no LinkedIn MCP), do NOT start per-contact processing — follow the **Discovery Source Availability & No-Fabrication Rule** (defer real people, skip non-persons/unresolvables, write no facts). If ≥1 source is live, proceed with the fallback chain in `references/discovery-fallback.md`. 3. **Clear pre-existing garbage** — scan for known junk values before enriching so COALESCE preserves nothing 4. placeholder — the `parents[2]` path bug in scripts can create stale DB files at `/commons/db/ocas-weave/weave.sqlite`, `/profiles/commons/db/ocas-weave/weave.sqlite`, and `/profiles/indigo/skills/commons/db/ocas-weave/weave.sqlite`. Only the canonical path (`/profiles/indigo/commons/db/ocas-weave/weave.sqlite`) is correct. Stale DBs confuse subagent enrichment writes. Remove them before enriching. 5. **Check edges FK constraint** — run `python3 -c "import sqlite3; c=sqlite3.connect('/profiles/indigo/commons/db/ocas-weave/weave.sqlite'); r=c.execute('PRAGMA foreign_key_list(edges)').fetchall(); print(r)"`. If `target_id` references `persons(id)`, run `python3 scripts/migrate_edges_fk.py` before enriching. The `target_id` column is polymorphic (can point to `facts.id` or `preferences.id`) — a wrong FK causes `HasFact` edge inserts to fail silently. 6. Query contacts with gaps 7. Process each contact through Scout → Sift → Sherlock → Write 8. Run periodic Google sync after every 10 enriched contacts 9. Final Google sync ### Unresolvable Contacts Protocol Read `references/unresolvable-contacts.md` for the full decision flow. Key rule: when a contact's name is common and no email/phone/location disambiguates, **skip** — never guess a match. Log to `decisions.jsonl` with reason `skip_unresolvable`. **Script note**: Shared enrichment extraction, search, and validation logic lives in `scripts/weave_enrich.py`. Both `quick_enrich.py` and `overnight_enrichment.py` import from it. To run overnight enrichment: `python3 overnight_enrichment.py`. To get contacts with gaps, query the Weave DB directly (see `references/query_patterns.md`). To check SearXNG health, use the diagnostic curl in `enrichment-pipeline.md`. For manual enrichment of high-value contacts, use the full quality pipeline: 1. **Pre-Search Seed Quality Check** — Check `source_type` and `confidence`. 2. **Read** — Query Weave for the existing Person record. 3. **Search** — Use SearXNG for identity-resolved research. 4. **Enrich (Scout → Sift → Sherlock)** — Full pipeline. 5. **Write** — MERGE on Person, CREATE on Fact/Preference. Always read back. 6. **Search again** — Follow-up with enriched data. 7. **Sync** — After writes touching mapped fields, sync to Google Contacts. ## Google Contacts sync See `references/connectors.md` for full sync rules. Key points: - Match by `google_resource_name`, then email, then phone. Never match on name alone. - Gap-fill only — Weave provenance wins conflicts. - Outbound requires `writeback.google_contacts: true` (the default) AND a previous sync checkpoint. ## Recovery behavior See `references/recovery-weave.md` for the full recovery contract. ## Discovery Source Availability & No-Fabrication Rule Every enrichment run depends on a working Scout source. In agent/cron context these sources are external and can fail as a group. **There is no point in the pipeline where fabricating occupation/org data is acceptable.** If you cannot discover real data, you MUST NOT write it. **At pipeline start, run `python3 scripts/discovery_probe.py`** to test which sources are live. Interpreting results: - `web_search` (MCP) — verify in-session it returns non-empty `data.web`. Empty `success:true` payloads = non-functional; treat as down. - SearXNG — see `unresponsive_engines`. `brave: too many requests` = **rate-limited, recovers after backoff** (space queries 15–20s apart; retry with `2**(n+1)*5`s sleeps). `karmasearch: access denied` = needs re-grant, won't recover this run. A single 0-result response is NOT proof of death — a later query can return results. - DuckDuckGo HTML — last-resort discovery (curl + regex on `result__a`); hard rate-limits to an HTTP 202 anomaly page after a few queries. **Never burn DDG on test queries.** See `references/discovery-fallback.md`. **Decision matrix when Scout is degraded/unavailable:** | Situation | Action | |---|---| | ≥1 source live | Proceed with fallback chain (LinkedIn MCP → web_search → SearXNG → DDG). | | ALL sources down | Do **not** run Sift/Sherlock on zero input. Instead: (a) skip non-person/business + unresolvable contacts normally; (b) **defer** real people who already carry recoverable Google-sync context (missing only one field) — they are queued, not dropped; (c) log a `pipeline_blocked` decision with `"stage":"scout"` and the unavailable-source list; (d) write **NO** enrichment facts. | | Some contacts resolvable, others not | Enrich the resolvable ones; defer/skip the rest per `references/unresolvable-contacts.md`. | **No-fabrication is non-negotiable:** an empty discovery result means "no data," never "write a best guess." Reporting a blocked run honestly (and deferring real people) is the correct outcome. See `references/discovery-fallback.md` for the full fallback playbook and `scripts/discovery_probe.py`. ## Constraints See `references/constraints.md` for the full constraint set. ## Gotchas See `references/gotchas-weave.md` for the full gotcha catalog including: - SQLite WAL mode and concurrent access - Google OAuth token handling and cross-account contamination - LinkedIn profile fetching - Python environment (no liblbug.so needed) - **Cron mode constraints and workarounds**: `execute_code` is BLOCKED in cron jobs — it runs arbitrary local Python including subprocess calls that bypass approval. Use `terminal` with inline `python3 -c "..."` or `python3 /path/to/script.py` for all Python operations. Set `timeout` appropriately (max 600s for foreground). Background processes with `notify_on_complete=true` work for long sync jobs. - Contact merge diagnosis and repair ## OKRs Read `references/okrs.md` for Weave-specific OKR definitions and targets. ## Optional skill cooperation - Chronicle — read for entity enrichment; entity observations emitted via journal payloads - Scout — receive OSINT findings about people as upsert candidates - Dispatch — provide social graph context for communication drafting - Clay (Mesh MCP) — CRM sync via Smithery ## Journal outputs - Observation Journal — query runs, upsert runs, import runs - Action Journal — sync runs, writeback runs ## Initialization On first invocation, `_open_db()` handles auto-initialization. See `references/init_pattern.md`. ## Background tasks | Job name | Schedule | Command | |---|---|---| | `weave:update` | `0 0 * * *` | `weave.update` | | `weave:sync-google` | `0 4 * * *` | `AGENT_ROOT=/profiles/indigo HOME=/root python3 -u {skill_root}/scripts/google_sync.py` | | `weave:enrichability-recalc` | `0 1 * * *` | `python3 {skill_root}/scripts/recalculate_enrichability.py` | ## ⚠️ CRON INVOCATION: IGNORE THE RUNBOOK IN THE MESSAGE **The cron job's user message often contains a hardcoded pipeline runbook that is STALE.** It may reference removed components (`enrichment_data.py`, LadybugDB bridge, `systemctl stop ladybug-bridge-weave.service`). **ALWAYS defer to this skill's own documentation** over the runbook in the invocation message. The skill is updated first; the cron message template lags by weeks or months. Specific runbook instructions to IGNORE: - "Stop the LadybugDB bridge" → **NOOP** (bridge removed June 2026) - "Run enrichment_data.py" → **DOES NOT EXIST** (use WeaveDB queries directly) - "Run google_sync.py without AGENT_ROOT" → **WILL FAIL** (must set `AGENT_ROOT=/profiles/indigo HOME=/root`) - "Restart bridge after" → **NOOP** (bridge removed) - Any step using `execute_code` → **BLOCKED IN CRON** (use `terminal` + temp file instead) ## Agent-Driven Overnight Enrichment When the enrichment pipeline is run as a cron job (agent-driven, not script-driven), the agent has access to all MCP tools including web_search, web_extract, Composio LinkedIn, and SearXNG. In this mode: ### Script Dependencies - **`enrichment_data.py` does NOT exist on disk.** Do not attempt to run it. Instead: - For SearXNG health: `curl -s "http://localhost:8888/search?q=test&format=json&limit=3"` - For contacts with gaps: query WeaveDB directly (`SELECT p.id, p.name, ... FROM persons p LEFT JOIN edges e ... HAVING occupation IS NULL OR org IS NULL`) - For writing enrichment: use `WeaveDB.execute_write()` directly — see **Enrichment Write Pattern** below - For stats: query WeaveDB directly ### Enrichment Write Pattern (Agent-Driven) The WeaveDB schema requires three operations per contact. The `facts` table has NO `person_id` column — linkage is via the `edges` table. **Schema reference:** - `persons`: id, name, email, phone, location_city, location_country, occupation, org, google_resource_name, clay_id, source_type, source_ref, confidence, record_time, valid_from, valid_until - `facts`: id, predicate, value, confidence, source_type, source_ref, record_time (NO person_id) - `edges`: id, source_id, target_id, rel_type, strength, since, context, source_ref, confidence, record_time **Write pattern (use `terminal` with inline Python — `execute_code` is BLOCKED in cron mode):** ```python import sys, json, uuid sys.path.insert(0, '/profiles/indigo/skills/ocas-weave/scripts') from weave_sqlite import WeaveDB from datetime import datetime, timezone db = WeaveDB() now = datetime.now(timezone.utc).isoformat() # For each contact: # 0. READ CURRENT STATE first (only update NULL/empty fields) current = db.execute("SELECT id, name, occupation, org FROM persons WHERE id = ?", (person_id,)) if not current: skip person = current[0] # 1. Build UPDATE dynamically — only set fields that are NULL or empty string update_fields, update_vals = [], [] if occupation and not person.get('occupation'): update_fields.append("occupation = ?"); update_vals.append(occupation) if org and not person.get('org'): update_fields.append("org = ?"); update_vals.append(org) if not update_fields: skip # nothing to update db.execute_write( f"UPDATE persons SET {', '.join(update_fields)} WHERE id = ?", tuple(update_vals + [person_id]) ) # 2. INSERT fact (the enrichment payload) fact_id = str(uuid.uuid4()) db.execute_write( 'INSERT INTO facts (id, predicate, value, source_type, source_ref, confidence, record_time) VALUES (?, ?, ?, ?, ?, ?, ?)', (fact_id, json.dumps({"occupation": occupation, "org": org, "confidence": confidence}), source_type, source_ref, confidence, now) ) # 3. INSERT edge linking person → fact edge_id = str(uuid.uuid4()) db.execute_write( 'INSERT INTO edges (id, source_id, target_id, rel_type, source_ref, confidence, record_time) VALUES (?, ?, ?, ?, ?, ?, ?)', (edge_id, person_id, fact_id, 'HasFact', source_ref, confidence, now) ) # 4. READ-BACK VERIFY verify = db.execute("SELECT occupation, org FROM persons WHERE id = ?", (person_id,)) assert verify[0]['occupation'] == occupation or verify[0]['org'] == org, "WRITE FAILED" ``` **Why not COALESCE?** The skill's earlier pattern used `COALESCE(?, occupation)` which works but makes it impossible to detect "nothing changed" — you always write a fact row even if the data was identical. The explicit state-check pattern above avoids redundant fact writes and makes debugging easier. **Important:** Skip contacts where both `occupation` AND `org` are null AND confidence < 0.5 — no meaningful data to write. ### LadybugDB Bridge - The `ladybug-bridge-weave.service` **no longer exists** after the SQLite migration (June 2026). Do not attempt to stop/start it. The SQLite backend does not require it. ### Google OAuth Failure Handling - If `google_sync.py` fails with HTTP 401 / `invalid_grant`, the refresh token has been revoked. **The script now exits with code 2 and a clean ABORT message** (no traceback). Log the failure and continue with enrichment using MCP tools. Do not halt the entire pipeline. must re-authorize OAuth manually — there is no programmatic workaround. - When Google sync is unavailable, enrichment data can still be gathered via web_search, web_extract, Composio web tools, and direct page fetching (curl + Jina Reader). ### Page Fetching - `web_extract` fails with SearXNG backend ("search-only backend cannot extract URL content"). Use `curl -s "https://r.jina.ai/URL"` for page content fetching instead. - LinkedIn profiles: direct HTTP with browser User-Agent works; Jina Reader is blocked for LinkedIn. Composio `LINKEDIN_GET_PERSON` requires a `person_id` (not username) — there is no name-search tool. ## Self-update `weave.update` pulls the latest package from GitHub. See `references/self-update.md`. ## Database maintenance See `references/database_maintenance.md`. See `references/graph-storage-backend-research.md` for the full evaluation of alternatives to LadybugDB (SQLite adjacency lists recommended). ## Pitfalls - **Never leave broken scripts after a migration**: When you migrate a shared backend (like LadybugDB → SQLite), you MUST update ALL scripts that depend on it in the same session. Do not wait to be told. Check `grep -rl "old_import" scripts/` and fix every hit. - **Fix known issues immediately without asking**: When you identify a problem and know how to fix it, apply the fix immediately. Do not ask "should I fix this?" or wait for the user to tell you. If something is broken and the fix is clear, just fix it. - **Do not ask confirmation on approved plans**: When the user says "yes" to a plan, execute immediately. Do not re-ask "should I proceed?" or present alternatives after approval. - **Module-level imports**: Python names imported inside `if __name__ == "__main__"` are NOT visible to module-level functions. Import `timedelta`, `sqlite3`, and all other dependencies at the top of the file. - **Shared auth module**: All Google OAuth + API call logic lives in `scripts/google_api.py`. Import from there — never duplicate `get_access_token`, `api_get`, `api_post`, or `api_patch` in individual scripts. The shared module handles token refresh, rate-limit backoff, and error handling consistently. - **SQLite FK constraints on polymorphic references**: The `edges.target_id` is a polymorphic reference (can point to `persons.id`, `facts.id`, or `preferences.id` depending on `rel_type`). Do NOT add `FOREIGN KEY (target_id) REFERENCES persons(id)` — it breaks `HasFact` and `HasPreference` edges. If the FK already exists in a live DB, use a migration script to recreate the table without it (see `scripts/migrate_edges_fk.py` for the pattern). - **Schema code vs live DB divergence**: `CREATE TABLE IF NOT EXISTS` in `weave_sqlite.py` only runs on first DB creation. Schema fixes in code do NOT apply to existing DBs. Always write an explicit migration script when changing DDL on a live database, and verify row counts before/after. - **Script name references**: When referencing other scripts in subprocess calls or Popen, verify the filename exactly matches what's on disk. `enrichment_control.py` referenced `overnight_weave_enrichment.py` but the actual file is `overnight_enrichment.py` — always `ls scripts/` to confirm. - **Enrichment write pattern consistency**: When writing enrichment data to Weave, always use `weave.execute_write()` / `weave.execute()` (the WeaveDB abstraction layer) rather than raw `sqlite3` connections. Raw connections bypass FK enforcement, skip WAL mode, and can leave the DB in an inconsistent state. The only exception is bulk import via `weave.bulk_import()` which manages its own connection lifecycle. - **Wrong `person_id` → silent 0-row UPDATE + edge FK failure + orphaned fact (REAL FAILURE MODE)**: In the three-step write, if the `edges` INSERT fails with `FOREIGN KEY constraint failed` but the preceding `persons` UPDATE raised no error, the cause is almost always a wrong `person_id` (e.g. a transposed UUID segment — `a1e3` vs `1ae3` copied from the gap-query output), NOT DB corruption. The UPDATE matched 0 rows (sqlite3 does NOT error on a 0-row UPDATE), the `facts` INSERT committed (no FK on `facts`), and only the edge's `source_id → persons(id)` FK caught it — leaving an orphaned fact with no edge. **Fix/avoid**: (1) copy IDs programmatically from the gap-query result, never by hand; (2) before the write loop, assert `SELECT id FROM persons WHERE id = ?` returns the row; (3) on FK failure, `DELETE FROM facts WHERE id = ?` the orphan, correct the ID, re-run the full three-step write, and confirm via read-back. Detected and recovered this way on the 2026-07-07 run (Contact B). - **Printed `None` from a query = SQL NULL, not the string `'None'`**: When inspecting `db.execute()` results in terminal output, a field shown as `None` is Python's `None` (i.e. SQL NULL), not the literal text `"None"`. A cleanup pass that matches `WHERE org = 'None'` matches nothing and wastes a cycle. Guard with `IS NOT NULL` and only treat a value as a string when `isinstance(v, str)`. Several contacts in 's graph displayed as `None` in output but were already NULL. - **Enrichment three-step write pattern**: Writing enrichment data requires THREE operations because the `facts` table has no `person_id` column. The linkage is via `edges`: (1) `UPDATE persons SET occupation=..., org=..., location_city=? WHERE id=?`, (2) `INSERT INTO facts (id, predicate, value, ...) VALUES (?, 'enrichment', ...)`, (3) `INSERT INTO edges (id, source_id, target_id, rel_type, ...) VALUES (?, person_id, fact_id, 'HasFact', ...)`. Do NOT try to insert `person_id` into `facts` — the column does not exist. - **Shared enrichment extraction**: All web scraping, content extraction, and field validation logic lives in `scripts/weave_enrich.py`. Both `quick_enrich.py` and `overnight_enrichment.py` import from it. When modifying extraction patterns (regex, validation rules, search queries), update `weave_enrich.py` — never edit duplicated copies in individual scripts. - **Post-migration reference drift**: After a backend migration (e.g., LadybugDB → SQLite), ALL reference files must be audited — not just code. `schemas.md`, `gotchas-weave.md`, `connectors.md`, and any file with code examples or schema docs will silently drift. Check every `.md` in `references/` for stale imports, old DB paths, deprecated query languages, and outdated CLI commands. Orphaned reference files (not linked from SKILL.md) should be archived or deleted. - **WeaveDB default path calculation**: In `scripts/weave_sqlite.py`, `AGENT_ROOT = Path(__file__).resolve().parents[2]` goes up 2 levels from the script, but the skill lives at `profiles/indigo/skills/ocas-weave/scripts/`. The correct path is `parents[3]` to reach `profiles/indigo/`. Using `parents[2]` points to `skills/` which has a stale/empty `commons/db/ocas-weave/weave.sqlite`. This causes silent write failures — the DB opens but has no data. Verify `DEFAULT_DB_PATH` resolves to the expected location on first import. - **Function signature drift in pipeline scripts**: `overnight_enrichment.py` called `sift_extract_from_pages(name, org, all_results, max_pages=3)` but the function signature is `sift_extract_from_pages(name, search_results, max_pages=3)`. The extra `org` argument shifted `all_results` into `max_pages` and `max_pages=3` was ignored. Always verify function signatures match when calling shared functions from `weave_enrich.py`. - **Enrichment field validation too permissive (PATCHED 2026-06-18)**: `validate_field()` in `weave_enrich.py` previously allowed garbage through: sentence fragments as occupations ("As the Editorial Director"), city names as org ("Chicago", "Los Angeles"), single-word generic orgs ("Professional", "Accidents", "per", "newsletter"), partial org names ("was", "Updates", "Product", "San"), invalid locations ("Teague, CP"), junk emails ("leaflet@1.9.4"), and duplicate fact inserts. **Fix applied**: org validation now rejects known STATIC_CITIES, generic non-company words, sentence fragments (was/were/been/have/has), values without uppercase letters, and single-character values. Occupation validation requires title-case tokens. Always deduplicate facts before insert. - **SearXNG connection resets under load**: During overnight enrichment, SearXNG can return `Connection reset by peer` or `Remote end closed connection without response` errors when hit with rapid sequential searches. Add retry logic with exponential backoff (3 attempts, 2s/4s/8s delays) to `searxng_search()` in `weave_enrich.py`. - **overnight_enrichment.py duplicate processing**: The script's progress tracking does not prevent re-processing contacts that were already enriched in a previous run. If interrupted and restarted, contacts appear in the progress file but may already have facts written. The script also writes duplicate facts (same predicate/value for the same person) when `enrich_weave_contact()` is called multiple times for the same contact. Always deduplicate after enrichment runs. - **Google outbound sync etag failures**: The outbound phase of `google_sync.py` frequently returns HTTP 400 with "person.etag is different than the current person.etag" for a subset of contacts (observed ~187/587, ~32%). This means Google's contact data was modified externally between the etag fetch and the update push. **Workaround**: The sync checkpoint prevents re-pushing previously successful contacts, so subsequent runs only retry the failed batch. If failures persist across runs, the checkpoint may need investigation. This is a known rate-limiting/consistency issue, not a data loss risk — inbound sync is unaffected. ## Agent-Driven Enrichment Pitfalls (June 2026) See `references/enrichment-agent-driven.md` for the full session write-up. Key takeaways: - **Jina Reader blocks LinkedIn**: `r.jina.ai/linkedin.com/in/...` returns `SecurityCompromiseError` (HTTP 451) with "Anonymous access to domain www.linkedin.com blocked." **Workaround**: Skip LinkedIn URLs entirely in the Sift phase. Use SearXNG result snippets (title + content) for extraction instead. - **Direct HTTP to LinkedIn returns authwall**: LinkedIn redirects to `linkedin.com/authwall` for unauthenticated requests. **Workaround**: Same as above — rely on search engine snippets. - **Regex extraction produces sentence fragments**: `extract_from_content()` in `weave_enrich.py` captures too much text as occupations (e.g., "Prior to Google, Blaise was a Distinguished Engineer", "I am a Senior Product Manager"). **Workaround**: Apply post-extraction cleaning: reject if starts with known bad prefixes ("I am", "Currently", "Prior to", "Leveraging", "Please send", "Show Details"), reject if >50 chars, reject if person's own name appears in the value, require title-case first letter. See the `clean_occupation()` pattern in `/tmp/batch_enrich_v2.py` for a working implementation. - **Wrong-person data on shared pages**: When fetching pages that mention multiple people (e.g., event pages, company team pages), regex extraction can capture another person's title/email. **Workaround**: Reject occupation values containing the contact's own name (means the regex captured a different person's context). Reject emails that don't match the contact's known domain or name pattern. - **Garbage org values from navigation/UI text**: Regex captures UI elements like "Pages", "Baseball", "Us", "El", "Save", "User" as org values. **Workaround**: Maintain a reject set of known garbage org values. Require org to be a proper noun (starts with capital letter, not a common English word). Reject single-character values. - **web_search as primary discovery tool**: `web_search` (Exa AI) returns higher-quality LinkedIn data than SearXNG for professional profiles. The LinkedIn title + description in search results is more reliable than regex extraction from full page HTML. **Recommendation**: Always run `web_search` first for each contact, use results for occupation/org/location extraction, then supplement with SearXNG for additional sources. **Key pattern**: Parse the LinkedIn title + description from web_search results directly — the format is typically "Job Title at Company | LinkedIn" with a description containing location. This avoids the Jina Reader page-fetch step entirely for most contacts. - **Composio LinkedIn requires person_id not username**: `LINKEDIN_GET_PERSON` takes `person_id` (e.g., `yrZCpj2Z12`), not vanity username (e.g., `jonesabi`). There is no name-search tool in the LinkedIn MCP. **Workaround**: Extract `person_id` from LinkedIn profile URLs (`linkedin.com/in/username` → use username to search via web_search, then extract the actual profile ID from the canonical URL or use the username directly with direct HTTP). - **Data quality red flags for org values**: Reject org values that are: (1) known city names, (2) single generic words (Professional, Employees, Newsletter), (3) sentence fragments containing verbs like "was"/"were"/"been", (4) values without any uppercase letters, (5) email addresses or URLs, (6) values matching the person's own name. - **google_api.py silent refresh failure (PATCHED 2026-06-20)**: `get_access_token()` in `google_api.py` had two compounding bugs: (1) the credential file stores `expiry` as a Unix timestamp float (e.g., `1781939144.66`) but the code called `datetime.fromisoformat(expiry)` which throws `ValueError` on a float; (2) the `except Exception: pass` silently swallowed the error and returned the expired token without refreshing. The Google People API then returns HTTP 401. **Fix**: check `isinstance(expiry, (int, float))` and use `datetime.fromtimestamp()` for numeric values, otherwise fall back to `fromisoformat()`. After fixing, also verify the refresh token itself hasn't been revoked — `invalid_grant` from the token endpoint means the OAuth consent flow must be re-completed by . - **google_sync.py unhandled auth failure (PATCHED 2026-06-20)**: Even after `get_access_token()` was fixed to raise `RuntimeError` on `invalid_grant`, the `google_sync.py` `__main__` entry point had no try/except — it let the exception propagate as a raw traceback and exit code 1. Cron jobs should never crash with tracebacks. **Fix**: wrap `main()` in a try/except that catches `RuntimeError` containing "refresh token revoked" and exits with code 2 and a clean `ABORT` message to stderr. This distinguishes auth failures (exit 2) from other crashes (exit 1) and avoids noisy cron alerts for a known unrecoverable state. - **Credential file managed by MCP server**: The file at `/credentials/.json` is written by the google_workspace MCP server, which may overwrite `expiry` back to a float after a refresh. Always handle BOTH float and ISO format in `get_access_token()`. If the MCP server overwrote the file between your `json.dump` and the next read, the fix is still safe because it handles both formats. - **`enrichment_data.py` does not exist**: There is no `enrichment_data.py` on disk. Use direct WeaveDB queries and `curl` for SearXNG health. See the "Agent-Driven Overnight Enrichment" section above. - **`web_extract` cannot fetch URLs with SearXNG backend**: Use `curl -s "https://r.jina.ai/URL"` instead. This is the reliable page-fetching method in cron/agent context. - **LadybugDB bridge removed**: `ladybug-bridge-weave.service` no longer exists. Skip stop/start bridge steps in the enrichment pipeline. - **WeaveDB.execute() returns dicts, not tuples**: `db.execute()` returns `list[dict]`, not `list[tuple]`. Use `r[0]['column_name']`, NOT `r[0][0]` — the latter raises `KeyError: 0`. - **Cron invocation may pass a stale runbook**: See the **⚠️ CRON INVOCATION** section at the top of this skill. The cron job's user message sometimes includes a hardcoded pipeline runbook that references removed components (`enrichment_data.py`, LadybugDB bridge). **Always defer to the skill's own documentation** over the runbook in the invocation message. The skill is updated first; the cron message template may lag. If the runbook says "stop the LadybugDB bridge" or "run enrichment_data.py", those instructions are stale — skip them and follow the skill's Agent-Driven Enrichment section instead. - **Heredoc Python in terminal triggers false backgrounding detection**: Using `python3 << 'EOF'` in a foreground `terminal()` call may be rejected with "Foreground command uses '&' backgrounding". **Workaround**: Write the script to a temp file (`/tmp/weave_batch_enrich.py`) via `write_file`, then run it with `python3 /tmp/weave_batch_enrich.py`. - **Pre-existing garbage data in persons table**: Some contacts have junk occupation/org values from prior bad enrichment runs (e.g., occupation="Save", org="Riegel", org="New", org="St", org="YouTube", org="PI", org="_VOIS", occupation="gram Manager Big Tech Refuge", org="George", org="Donna Karan New York"). Before enriching, scan for and clear known garbage values so COALESCE doesn't preserve them. Common garbage: single-word orgs that aren't companies ("New", "St", "Early", "Los", "Experienced", "Arsenal", "PI", "Converge", "DockerCon", "YouTube", "George"), non-job occupations ("Save", "All Restaurants", "Short Interest", "Building Manager", "gram Manager Big Tech Refuge"), brand-orgs that aren't the person's employer ("YouTube", "Donna Karan New York"), partial org names ("_VOIS"). - **Stale `org=Google` from bad enrichment**: Many contacts got `org=Google` from sync metadata or prior enrichment. **Clearing heuristic**: keep `org=Google` only if the person has corroborating data — either an `@google.com` email address OR both occupation AND location_city populated. Without corroboration, set org to NULL. Same heuristic applies to other major tech companies (Microsoft, Salesforce, Amazon) when there's no email match or other data to confirm. - **Non-person entries in contacts**: Some "persons" are actually businesses/services (Doordash, Amazon.com, Resy, Visualping, Wealthfront, Harborworks Studio). Skip these during enrichment — they have business emails (info@, support@) and no individual professional profile. - **sys.path must use absolute paths in cron scripts**: When writing batch scripts to `/tmp/`, use `sys.path.insert(0, '/profiles/indigo/skills/ocas-weave/scripts')` — NOT a relative path like `'scripts'`. The cron working directory is the home dir, not the skill dir. Relative paths cause `ModuleNotFoundError`. - **Empty string vs NULL**: The persons table uses both `NULL` and `''` (empty string) for unfilled fields. Your update filter must check BOTH: `if not person.get('occupation')` catches both None and '' in Python. Don't write separate SQL for `IS NULL` and `= ''`. - **Duplicate person records**: Some names appear multiple times with different IDs (e.g., two "Contact C" records, two "Contact A" records). Query by name to find all variants and enrich each one. Don't assume ID uniqueness by name. Note: dual-person queries in cron-pipeline-runbook.sql LIMIT 50 may return duplicates that inflate coverage metrics — track by distinct name, not distinct ID, when reporting "both occ+org" counts. - **Subagent enrichment writes may hit wrong DB path**: When using `delegate_task` to spawn enrichment subagents, the subagent receives NO context about the correct DB path by default. If the subagent uses `WeaveDB()` (which resolves via `parents[3]`) it lands correctly. But if it uses `sqlite3.connect()` directly or imports via a relative `sys.path.insert(0, 'scripts')`, it may hit `/commons/db/ocas-weave/weave.sqlite` (stale, 953 persons) instead of `.../profiles/indigo/commons/db/ocas-weave/weave.sqlite` (canonical, 1052 persons). This produces enrichment facts in the wrong DB that are invisible from the canonical one. **Fix**: Always include `canonical_db_path = '/profiles/indigo/commons/db/ocas-weave/weave.sqlite'` in subagent task context. After subagent completion, verify enrichment facts by ID in the canonical DB. ## Support File Map | File | When to read | |------|-------------| | `references/schemas.md` | Before any DDL, upsert, or import — Python usage pattern and schema | | `references/gotchas-weave.md` | Before any Weave operation — full gotcha catalog | | `references/query_patterns.md` | Before any weave.query call — SQL templates for all modes | | `references/connectors.md` | Before any Google/Clay sync | | `references/sqlite-backend-research.md` | Storage backend details, migration notes, SQLite schema | | `references/enrichment-pipeline.md` | Overnight enrichment architecture, SearXNG retry pattern | | `references/enrichment-agent-driven.md` | Agent-driven overnight enrichment pipeline architecture, cleaning rules, tool workarounds (June 2026) | | `references/enrichment-run-2026-06-30.md` | **Session write-up for June 2026 overnight run** — what worked, what didn't, contacts enriched, action items for next run | | `references/enrichment-write-pattern.md` | **Exact SQLite write pattern for agent-driven enrichment** — three-step (persons UPDATE → facts INSERT → edges INSERT), cron-mode terminal usage, read-back verification | | `references/cron-pipeline-runbook.md` | **The correct step-by-step runbook for agent-driven enrichment** — modern pipeline (no LadybugDB bridge, no enrichment_data.py), cron-mode terminal usage, confidence scoring guide, read-back verification pattern | | `references/constraints.md` | Full constraint set | | `references/config-defaults.md` | Default config structure | | `references/self-update.md` | Self-update procedure | | `references/enrichment-data-quality.md` | Data quality patterns, garbage categories, validation rules, SearXNG reliability | | `references/unresolvable-contacts.md` | **Unresolvable contacts protocol** — when to skip (common name, no disambiguator, multiple conflicting profiles), identity resolution ladder, confidence thresholds, log format | | `references/recovery-weave.md` | Recovery contract details | | `scripts/weave_sqlite.py` | SQLite backend module — import `WeaveDB` from here | | `scripts/google_api.py` | Shared Google OAuth + API helpers — import `get_access_token`, `api_get`, `api_post`, `api_patch`, `PEOPLE_API_BASE` from here. All scripts that talk to Google APIs should use this module, not duplicate auth logic. …(truncated)