AI Session Watch

ai-session-watch

Project-local instinct learning for ai-engineering with two modes: passive observation during a session, and active consolidation on demand. No daemons or background workers -- the LLM itself is the observer.

Workflow

1. Observation is always-on via the instinct-observe.py hooks (PreToolUse + PostToolUse) — no manual activation, and /ai-start does NOT start it. Invoking bare /ai-session-watch is optional and only emits a one-line acknowledgement.
2. As work happens, the model passively notes corrections, error recoveries, and skill-invocation sequences.
3. Before a commit or PR, invoke /ai-session-watch --review to run the 5-step consolidation: extract → enrich → write → evaluate → create work items.
4. High-confidence proposals (>= 0.7, evidence >= 3) auto-generate work items via gh issue create or az boards work-item create.

Artifact Set

Supported Families (v2)

The canonical store supports only these sections:

• corrections -- user corrections to AI behavior (LLM-extracted during --review)
• recoveries -- error recovery patterns (hook-detected + LLM-enriched)
• workflows -- skill invocation sequences (hook-detected + LLM-enriched)

Anything outside those families is out of scope.

Commands

/ai-session-watch (listening mode)

Enter passive observation mode. Output ONLY this single line, then go silent:

observing the session...

Do nothing else — no file reads, no analysis. The LLM passively notes corrections, recoveries, and workflow patterns; consolidation happens only on --review.

/ai-session-watch --review (consolidation)

5-step consolidation (extract → enrich → write → evaluate → create work items). Run this before /ai-commit or /ai-pr to capture learnings.

Step 1: EXTRACT

Review the current conversation for:

• Corrections: moments where the user corrected AI behavior, rejected an approach, or redirected a decision.
• Recoveries: errors encountered and how they were resolved.
• Workflows: skill invocation sequences and tool usage patterns.

For each observation, identify:

• relatedSkill -- which skill was active (e.g., ai-code, ai-test)
• diagnostic -- the error message or correction signal
• skillIssue -- what the skill got wrong or could improve

Step 2: ENRICH

1. Read .ai-engineering/state/observation-events.ndjson for hook-detected recoveries.
2. Read .ai-engineering/state/framework-events.ndjson for skill_invoked events to detect workflow patterns.
3. For each extracted observation, add semantic fields:
   • trigger -- what condition causes this pattern (e.g., "user says 'no, do X instead'")
   • action -- what the AI should do differently (e.g., "prefer X over Y in this context")

Step 3: WRITE

Upsert entries into .ai-engineering/observations/observations.yml using the v2 schema. Each family entry shares: pattern, trigger, action, relatedSkill, confidence (0.0-1.0), evidenceCount, domain (project|stack|team), lastSeen (ISO 8601). corrections and recoveries add diagnostic + skillIssue.

schemaVersion: "2.0"
corrections:
  - pattern: "<description>"
    trigger: "<what triggers this>"
    action: "<what to do>"
    relatedSkill: "<skill-name>"
    diagnostic: "<error or correction signal>"
    skillIssue: "<what the skill got wrong>"
    confidence: 0.7
    evidenceCount: 3
    domain: "project"
    lastSeen: "2026-04-27T00:00:00Z"
recoveries:
  - pattern: "<description>"
    # ...same fields; trigger/action/diagnostic describe error pattern + recovery steps + error message
workflows:
  - pattern: "<description>"
    # ...same fields except diagnostic/skillIssue; trigger/action describe the sequence

Merge rules:

• If an existing entry matches the same pattern (fuzzy match on trigger + action), increment evidenceCount and update lastSeen. Match if the trigger and action describe the same behavioral pattern using different wording. When in doubt, increment the existing entry rather than creating a duplicate.
• Apply confidence scoring: confidence_for_count(evidenceCount) yields 0.3/0.5/0.7/0.85 at thresholds 1/2/3/5+.
• Drop entries with confidence below 0.2 (decay threshold).
• Update .ai-engineering/observations/meta.json with new checkpoint. MUST stamp lastReviewedAt (ISO 8601, now) on a successful review (spec-165 D-165-05) — the System-B checkpoint the SessionStart observation-nudge reads to decide whether the backlog is stale (distinct from System-A lastExtractedAt, owned by the Stop-hook instinct extractor).

Step 4: EVALUATE

Cross-reference the updated instincts with project knowledge to produce actionable proposals:

1. Read .ai-engineering/LESSONS.md to check for already-captured patterns.
2. Read project context: CONSTITUTION.md, .ai-engineering/manifest.yml, and the target artifact (e.g., .claude/skills/ai-<skill>/SKILL.md or .claude/agents/ai-<agent>.md) to understand the improvement surface. If only .ai-engineering/CONSTITUTION.md exists, use it as a compatibility fallback.
3. Filter instincts: only those with confidence >= 0.7 AND evidenceCount >= 3 qualify as proposals.
4. For each qualifying instinct, check if the pattern is already captured in LESSONS.md -- if so, skip.
5. Append a new PROP-NNN entry to .ai-engineering/observations/proposals.md:

## PROP-NNN: <title>

- **Status**: proposed
- **Source**: <family> instinct, confidence <N>, evidence <N>
- **Pattern**: <what was observed>
- **Diagnostic**: <error message or correction signal from the instinct>
- **Proposed fix**: <specific change: update SKILL.md procedure, add LESSONS.md entry, adjust manifest config, etc.>
- **Target**: LESSONS.md | SKILL.md (<which skill>) | agent.md (<which agent>) | manifest.yml | hook (<which hook>)
- **LESSONS.md cross-ref**: <"none" or the matching lesson heading if partial overlap exists>

Number proposals sequentially (PROP-001, PROP-002, ...). Check existing entries in proposals.md before assigning the next number.

Step 5: CREATE WORK ITEMS

If .ai-engineering/manifest.yml has a work_items section, create trackable work items for each new proposal. Follow the same fail-open protocol as /ai-board sync: never block the calling workflow.

1. Read work_items.provider from .ai-engineering/manifest.yml.
2. Check for duplicates before creating:
   • GitHub: gh issue list --label "ai-engineering,instinct" --state open --json title
   • Azure DevOps: az boards query --wiql "SELECT [System.Title] FROM WorkItems WHERE [System.Tags] CONTAINS 'instinct' AND [System.State] <> 'Closed'" -o json
   • If a work item with a matching title already exists, skip and note "duplicate" in the output.
3. Create the work item:
   • GitHub (work_items.provider: github): gh issue create --title "instinct: [target] - [diagnostic]" --body "<proposal body>" --label "ai-engineering,instinct"
   • Azure DevOps (work_items.provider: azure_devops): az boards work-item create --title "instinct: [target] - [diagnostic]" --description "<proposal body>" --type "Task"
4. Update the proposal entry in .ai-engineering/observations/proposals.md: set Status to work-item-created and append - **Work item**: <ref> (e.g., #45 or AB#100).
5. If no work_items section in manifest, skip silently.
6. Fail-open: if CLI is not authenticated, project is not found, or command fails -- log a warning with remediation hint (e.g., gh auth login) but do not block the review.

Review-Mode Output

Structured summary: observations extracted (count per family), entries upserted (new vs. updated), proposals generated (count, titles), work items created (count, links). If no meaningful observations: "No consolidation needed -- session had no corrections, recoveries, or notable workflow patterns."

Boundaries

• Project-local only. No global instinct scope.
• One canonical observations.yml, not one file per instinct.
• Never store transcripts, prompts, responses, or raw tool payloads.
• Do not create instincts outside .ai-engineering/observations/.
• Do not invent unsupported pattern types beyond corrections/recoveries/workflows.
• Do not claim the system supports promotion, evolution, or global libraries.

Examples

Example — consolidate before committing

User: "review what was learned in this session before I commit"

/ai-session-watch --review

Runs the 5-step consolidation: extract → enrich with hook events → upsert into observations.yml → evaluate against LESSONS.md → create work items for high-confidence proposals. (Bare /ai-session-watch just prints observing the session... and stays silent until --review.)

Integration

Called by: user directly at session start. Calls: gh issue create / az boards work-item create (Step 5 work-item creation). Reads: decision-store.json, LESSONS.md, observations.yml, proposals.md. See also: /ai-learn (cross-session retro), /ai-skill-improve (acts on high-confidence proposals).

$ARGUMENTS