Research Pipeline Ops

research-pipeline-ops

IDENTITY: Ops.Runner. DetectCategories→CompareDates→Ingest→VerifyBatch→Report. Complements team-wiki/research-pipeline (the ingestion contract — threshold gate, frontmatter, contradiction protocol, index/log rules). This skill owns the run orchestration and verification around that contract: deciding which categories need processing, verifying a multi-page batch afterward, and the tool pitfalls that appear when executing in cron. Load both for a full pipeline run.

WHENUSE: Overflow pass over all research categories|Knowledge Manager Sunday ingest|Verifying a multi-page pipeline run|Any cron wiki batch hitting tool quirks. NoSkip: FindingsFileListing|LogDateParsing|RawImmutabilityCheck|WikilinkResolutionCheck.

Multi-category overflow detection

When the task is an overflow/KM pass over all research categories (not one assigned category):

1. List findings files: search_files(target='files', pattern='findings-*.md', path='raw/articles/research/'). Group by category directory.
2. Parse log.md for the last processed date per category: search for ## YYYY-MM-DD <slug> pipeline and overflow pass entries (they record the scope). A category with no such entry has never been processed.
3. Process only categories whose newest findings-YYYY-MM-DD filename date > last processed date. Equal-or-older findings are skipped. Retired categories (protocol pivot table, e.g. 2026-08-04) are skipped unless a newer findings file exists — a retired cron that still fires is an attention item, not a re-run trigger.
4. Categories with no findings files at all are skipped but MUST appear in the report ("no findings files") — a new active category silently producing nothing is a cron-health attention item.
5. Log: ONE batch entry with a ### <category> pipeline subsection per processed category plus a ### Summary block (precedents: 2026-08-02 and 2026-08-09 log entries). Never one entry per category.

Batch verification

For multi-page runs, run scripts/verify-pipeline-run.sh <wiki_root> <new page paths...> instead of hand-checking each page. It checks:

• frontmatter completeness (title/type/created/updated/tags/sources/workflow/confidence)
• ≥2 outbound [[wikilinks]] per page, all resolving to existing slugs
• raw/ immutability via mtime (no raw file modified on run date)
• index.md contains at least one entry per new page slug

Exit 0 = all pass; non-zero with a per-file failure list otherwise. Trust the script's failures — they are real (missing frontmatter field, unresolved link, raw write). Fix and re-run.

Full-wiki sweep (audit, not batch verify)

When the ask is "sweep / audit the wiki" (not verifying one batch), run scripts/wiki-sweep.py <wiki_root> — it audits every knowledge page in one read-only pass and prints a sectioned report. Checks (mirrors team-wiki/maintain but executable):

• frontmatter: required fields, type↔directory match, workflow presence in EITHER style, valid confidence, contested⇒contradictions, tags in taxonomy
• wikilinks: graph resolution, ghost-note inventory with backlink counts (promote at 3+), stale zero-backlink ghosts
• orphans: knowledge pages with no inbound links (flags [index-only] vs fully disconnected)
• index completeness: every knowledge slug in index.md AND every index link resolving to an existing page
• outbound minimum (<2 links), oversized pages (>200 lines per SCHEMA), raw/ mtime within 3 days, log entry count vs 500 threshold

Add the cron-drift cross-check to every sweep: compare cronjob list against the category table in operational/protocols/research-pipeline-categories.md, BOTH directions. Verified 2026-08-10: the 2026-08-04 pivot retired 6 categories (llm-agents, agent-protocols, context-engineering, local-inference, prompt-engineering, research-methodologies) and added 5 new ones (local-llm, open-source-models, generative-media, llm-research, ai-advancement), but ALL 6 retired crons remained scheduled/enabled (llm-agents fired the morning after its own log flagged retirement) and NO cron existed for any new category. A retired cron firing is an attention item, not a re-run trigger; an active category with no cron and no findings files is silent cron-health drift. Also check the inverse: a findings-YYYY-MM-DD.md with no registered cron that could have produced it is an anomaly worth flagging.

Workflow-tag convention split (verified 2026-08-10 sweep)

The wiki has TWO coexisting workflow styles, and SCHEMA.md, the research-pipeline skill, and verify-pipeline-run.sh each imply a different one:

• Key style (~103 pages): separate frontmatter key workflow: developing|stable|seedling|.... This is what verify-pipeline-run.sh greps for (^workflow:), so key-style pages pass verification.
• Tag style (newest pages, ~11): workflow:seedling INSIDE tags: [..., workflow:developing] (airi-ai-companion, streamcore-server, browsafe, solar/ pages). These FAIL verify-pipeline-run.sh's workflow: key check despite being valid per SCHEMA's "Workflow Tags" section.

Before treating a tag-style page as a verification failure, check for workflow:<state> inside the tags list. Also flag non-canonical values (workflow: proposed, workflow: reference seen on hermes-platform-tool-loading, hermes-desktop-profile-management, threejs-hologram-particle-techniques — allowed set is seedling|developing|stable|needs-review|stale) and pages with NEITHER style (moonlake-blender-mcp-ue5-pipeline-report, ue-ai-coding-pipeline-report — also missing updated/sources, non-SCHEMA type research-report).

RESOLVED 2026-08-10 (user approved "go with your recommendations"): the dedicated workflow: key is the canonical lifecycle form; in-tags workflow:xxx is legacy-accepted and normalized to the key on lint. SCHEMA.md now documents the key as canonical (frontmatter example, Field Reference, Workflow Tags section), verify-pipeline-run.sh validates the value against seedling|developing|stable|needs-review|stale, and 11 tag-style pages were normalized to key-style. When sweeping now: flag pages with NEITHER style or non-canonical values; converting tag-style → key-style is the sanctioned fix (moves workflow:seedling out of tags: into a workflow: key line). Also: type: summary is a VALID type for pages in concepts/ (research-report summaries, e.g. the two fixed pipeline reports) — do not flag it as a dir mismatch. See references/2026-08-10-sweep-remediation.md for the full remediation playbook.

Tool pitfalls (verified in the 2026-08-09 overflow run; sweep pitfall added 2026-08-10)

• BSD grep "invalid character range": grep -o '\[\[[^]]*\]\]' fails on macOS. Extract wikilinks with grep -oE '\[\[[^]]+\]\]' | tr -d '[]' | sort -u.
• read_file binary false positive: read_file may report "Binary file - cannot display as text" for a valid UTF-8 page (hit on concepts/model-landscape-2026.md). Verify with file <page> and xxd <page> | grep ' 00 ' — no null bytes means false positive. Read via cat, patch normally (patch still works), and do NOT rewrite the file to "fix" it.
• execute_code is denied in cron context (arbitrary Python needs user approval; none present in cron). Don't use it for batched searches in cron runs — batch parallel search_files/read_file calls instead, and use terminal for aggregate read-only verification sweeps.
• Tag-list parsing traps in sweep/audit scripts: naive regex token extraction ([a-z][a-z0-9-]+) splits workflow:seedling at the colon into two tokens (workflow, seedling) and mangles leading-digit tags (3d-modeling → d-modeling). On the 2026-08-10 sweep this produced ~90 false "no workflow tag" / "tag not in taxonomy" flags. Parse tags: by stripping brackets/quotes and splitting on commas, then check workflow:<state> as a single token. (The sweep script in scripts/wiki-sweep.py already does this.)

Protected-skill relationship

team-wiki/research-pipeline (and the team-wiki/* family) is user-owned in this profile: skill_manage refuses both the qualified name ("not found") and the unqualified alias ("not curator-managed — no usage record"). Until hermes curator adopt team-wiki/research-pipeline is run, this umbrella carries the ops knowledge. When the adoption happens, merge this skill's content into team-wiki/research-pipeline and delete this one (absorbed_into=team-wiki/research-pipeline).

Related

• team-wiki/research-pipeline — the ingestion contract this skill wraps
• team-wiki/maintain — daily lint/health-check (overlaps the verification sweep; curator may consolidate)
• note-taking/obsidian — file-tool discipline for vault work