Knowledge Pipeline

Knowledge Pipeline

When to Use

Use this skill when working on the repository's knowledge and learning flow across:

• scripts/knowledge/
• scripts/learning/
• docs/superpowers/

Typical tasks:

• build or refresh knowledge indexes
• synthesize archived material into reusable knowledge
• review or update open issue summaries
• maintain learning scripts that capture patterns from prior work
• debug the flow between knowledge extraction, learning, and documentation artifacts

Main Data Flow

source material / session artifacts
  -> scripts/knowledge/*
  -> synthesized knowledge / issue updates / index artifacts
  -> scripts/learning/*
  -> reusable lessons, summaries, and pipeline maintenance
  -> docs/superpowers/*
  -> human-readable designs, plans, and reference docs

Directory Roles

scripts/knowledge/

Primary operational scripts for knowledge handling. Common patterns include:

• build/update indexes
• query knowledge stores
• synthesize archive/history
• capture summaries
• update GitHub issues from structured knowledge

Relevant scripts on this machine include:

• build-knowledge-index.sh
• query-knowledge.sh
• synthesize_archive.py
• review-open-issues.py
• update-github-issue.py
• check-staleness.sh

scripts/learning/

Learning-stage automation and maintenance. Use this area when a workflow turns repeated observations into reusable process improvements.

Current anchor script:

• scripts/learning/comprehensive-learning.sh

docs/superpowers/

Design and planning documentation that explains the higher-level workflow and intended system behavior. Use this as the narrative/spec layer when code behavior and intended pipeline behavior drift apart.

Recommended Workflow

1. Inspect the current pipeline surface
   • list scripts under scripts/knowledge/ and scripts/learning/
   • read the closest matching design doc in docs/superpowers/
2. Identify the failing or missing stage
   • ingestion/indexing
   • archive synthesis
   • issue update
   • learning/summary generation
3. Run the narrowest script that reproduces the problem
4. Check generated artifacts and downstream consumers
5. Update tests or add a focused regression test
6. Re-run the stage and validate the next stage still receives the expected inputs

Validation Guidance

Prefer targeted tests where they exist. Examples in this repo include:

• scripts/knowledge/tests/test_review_open_issues.py
• scripts/knowledge/tests/test_synthesize_archive.py
• scripts/knowledge/tests/test_update_github_issue.py
• scripts/knowledge/tests/test-knowledge-scripts.sh

Run targeted validations such as:

uv run pytest scripts/knowledge/tests/test_review_open_issues.py -q
uv run pytest scripts/knowledge/tests/test_synthesize_archive.py -q
uv run pytest scripts/knowledge/tests/test_update_github_issue.py -q
bash scripts/knowledge/tests/test-knowledge-scripts.sh

Common Failure Modes

• index/build script succeeds but downstream docs are stale
• archive synthesis runs but outputs are not promoted or linked
• issue-update logic drifts from actual GitHub issue format
• learning scripts assume artifact locations that have moved
• docs/superpowers describes a pipeline no longer matching the code
• wiki health reports under knowledge/wikis/health-reports/ may be ignored by the repo-level /knowledge/wikis/* gitignore rule; when the task explicitly asks to commit generated health reports, stage them with git add -f knowledge/wikis/health-reports/<report-date>.{md,json} and verify with git status --short

Cron / Dirty-Workspace Pattern

When a scheduled job asks to run a knowledge script and commit generated artifacts, first check the live worktree state. If the current checkout is on a feature branch or has unrelated dirty files, create a temporary clean worktree from origin/main, run the script there, commit to main, push, then remove the temporary worktree. This avoids mixing generated reports with unrelated in-progress work while still satisfying direct-to-main cron tasks.

Troubleshooting Checklist

• verify the source artifact path first
• inspect one known-good output before changing code
• compare intended flow in docs/superpowers/ against actual script behavior
• keep code-stage and docs-stage fixes aligned
• add regression coverage when fixing a recurring pipeline break