Skill Functional Audit

/onex:skill_functional_audit — Functional Verification Audit

Skill ID: onex:skill_functional_audit Version: 1.0.0 Owner: omniclaude Created: 2026-04-14 Rationale: Addresses gap found in skill-catalog-2026-04-14.md — existence checks reported "0 broken" while 3 facades were live.

Why This Skill Exists

The prior skill-catalog-gap-sweep checked two things:

1. SKILL.md file exists in plugins/onex/skills/<name>/
2. Skill is registered in the plugin catalog

It did NOT verify:

• Whether backing handle() methods contain STUB / NotImplementedError markers
• Whether SKILL.md §Implementation Status declares phases as in-progress stubs
• Whether pure-instruction skills that describe complex orchestration have any executable backing
• Whether skills claiming node-backed dispatch actually have those nodes at the expected path

File existence ≠ functional. This skill closes that gap.

Announce at Start

"I'm using the skill_functional_audit skill to functionally verify all registered onex skills — checking backing nodes, stub markers, and invocability."

Phase 1: Enumerate Skills

Walk plugins/onex/skills/ and collect all skill directories (exclude _shared, _lib, _bin, _golden_path_validate):

ls "${ONEX_REGISTRY_ROOT}/omniclaude/plugins/onex/skills/" | \
  grep -v "^_\|__pycache__\|__init__\|progression.yaml\|slack-gate" | sort

For each skill directory, verify SKILL.md exists. Any missing SKILL.md = BROKEN verdict immediately.

Phase 2: Classify by Risk Tier

For each skill, classify into one of three tiers based on the skill name and SKILL.md category/tags fields:

WRITE/DISPATCH (skip live invocation, check backing)

Skills that create PRs, dispatch agent teams, trigger releases, or write to external systems:

epic_team, build_loop, release, redeploy, merge_sweep, overnight,
ticket_pipeline, ticket_work, create_ticket, create_followup_tickets,
plan_to_tickets, wave_scheduler, dispatch_worker, coderabbit_triage,
pr_polish, pr_review, pr_review_bot, auto_merge, dep_cascade_dedup,
dashboard_sweep, local_review, refill_sprint, generate_node,
adversarial_pipeline, multi_agent, integration_sweep, ticketing_epic_org,
linear_housekeeping, start_environment, record_friction, dod_sweep,
friction_triage, pipeline_fill, delegate, design_to_plan, checkpoint,
close-day, runner

INTERACTIVE (skip live invocation, check backing)

Skills requiring interactive session context or that modify session state:

session, onboarding, executing_plans,
resume_session, set_session, login, using_git_worktrees, demo,
systematic_debugging, writing_skills, insights_to_plan, authorize,
worktree, decompose_epic, ticket_plan, rrh

READ-ONLY (invoke if --skip-invocation not set, assert output schema)

Skills that read and report state without writes:

aislop_sweep, agent_healthcheck, baseline, bus_audit, ci_watch,
compliance_sweep, contract_sweep, coverage_sweep, data_flow_sweep,
database_sweep, decision_store, dispatch_watchdog,
dod_verify, duplication_sweep, env_parity, feature_dashboard, gap,
golden_chain_sweep, hook_health_alert, ticketing_insights, ticketing_triage,
observability, pipeline_audit, plan_audit, platform_readiness, pr_watch,
preflight, recall, rewind, runtime_sweep, tech_debt_sweep,
verification_sweep, verify_plugin

Default classification: If skill name doesn't match above lists, classify as INTERACTIVE (safer — skip invocation).

Phase 3: Backing Node Check (all tiers)

For each skill, determine its backing type:

3a. Explicit node-backed skills

Check if a node named node_<skill_name> exists at:

• omnimarket/src/omnimarket/nodes/node_<skill_name>/
• omnibase_infra/src/omnibase_infra/nodes/node_<skill_name>/
• omniclaude/src/omniclaude/nodes/node_<skill_name_orchestrator>/

If SKILL.md mentions a specific node name (grep: Backed by|backing node|node_), verify that node exists at the stated path.

3b. Stub marker scan

For all node-backed skills, scan handler files for stub markers:

grep -rn "raise NotImplementedError\|pass  # TODO\|# STUB\|STUB:\|Phase.*STUB\|TODO.*Implement" \
  <node_path>/handlers/*.py 2>/dev/null | grep -v __pycache__

Stub marker found = PARTIAL or STUB verdict (PARTIAL if some phases work, STUB if entire handle() is stubbed).

3c. Pure instruction check

If no backing node exists AND SKILL.md describes complex multi-phase orchestration with state management (dispatched.yaml, phase tracking, wave caps), classify as FACADE unless the skill is explicitly documented as "pure instruction" / "instruction-only" in its description.

Heuristic for FACADE detection:

• SKILL.md has > 200 lines AND describes stateful orchestration (state files, wave caps, in-flight tracking)
• AND no backing node named node_<skill_name> exists
• AND no run.py or executable script exists in the skill directory

Phase 4: Live Invocation (READ-ONLY skills only)

Unless --skip-invocation is set, invoke each READ-ONLY skill via the Skill tool with --dry-run if supported, or its lightest invocation mode:

Skill(skill="onex:<skill_name>", args="--dry-run")

Assert output contains at least one of:

• A status field (status, verdict, health, result)
• A count field (count, total, found, checked)
• A structured list or table

Timeout: 120 seconds per skill. If skill times out or throws an unhandled error: verdict = BROKEN.

Note: Skip invocation for skills known to require live infrastructure (Kafka, runtime host). Document these as SKIP (infra-dep) with the infrastructure dependency noted.

Phase 5: Produce Per-Skill Records

For each skill, produce a record:

- name: "onex:session"
  classification: INTERACTIVE
  backing_node: "node_session_orchestrator"
  backing_node_exists: true
  stub_detected: true
  stub_evidence: "handler_session_orchestrator.py:748 Phase 2 STUB, line 767 Phase 3 STUB"
  invocation_result: "SKIP (interactive)"
  linear_ticket: null
  verdict: PARTIAL
  notes: "Phase 1 implemented. Phase 2 (RSD scoring) and Phase 3 (dispatch) are explicit stubs."

Verdict values:

• WORKS — backing exists, no stubs, invocation passed (or SKIP with reason)
• PARTIAL — some phases implemented, others explicitly stubbed; tracked in Linear
• FACADE — skill describes complex logic but has no backing implementation; untracked
• STUB — entire handle() is stubbed (raise NotImplementedError or pass # TODO)
• BROKEN — SKILL.md missing, import error, invocation crashed

Phase 6: Write Audit Artifact

Write the full audit to disk:

TIMESTAMP=$(date -u +%Y%m%dT%H%M%SZ)
if [[ -z "${ONEX_STATE_DIR:-}" ]]; then echo "ERROR: ONEX_STATE_DIR must be set" >&2; exit 1; fi
OUTPUT_PATH="${ONEX_STATE_DIR}/skill-audits/${TIMESTAMP}.yaml"
mkdir -p "${ONEX_STATE_DIR}/skill-audits/"

Artifact schema:

generated_at: "2026-04-14T03:00:00Z"
skill_count: 91
verdicts:
  WORKS: 86
  PARTIAL: 2
  FACADE: 1
  STUB: 0
  BROKEN: 0
facade_skills:
  - "onex:pipeline_fill"
partial_skills:
  - "onex:session"
  - "onex:delegate"
skills:
  - name: "onex:aislop_sweep"
    classification: READ-ONLY
    backing_node: "node_aislop_sweep"
    backing_node_exists: true
    stub_detected: false
    invocation_result: "PASS"
    verdict: WORKS
  # ... all 91 entries

Also write a human-readable summary to ${ONEX_STATE_DIR}/skill-audits/latest-summary.md.

Phase 7: Fail Gate

If --fail-on-facade is true (default):

1. Collect all skills with verdict FACADE, STUB, or PARTIAL
2. Confirm the skill records include Linear ticket coverage in each linear_ticket field
3. If ANY facade/stub/partial has no linear_ticket: FAIL (exit nonzero)
4. If all facade/stub/partial skills have tracked Linear tickets: PASS (exit 0, log warning)

Failure message format:

SKILL AUDIT FAILED: 1 untracked facade(s) found

  FACADE: onex:pipeline_fill
    Evidence: No backing node node_pipeline_fill; SKILL.md describes stateful orchestration without implementation
    Action: File a Linear ticket and add its ID to the skill record

Run again after filing tickets to clear this gate.

Phase 8: Report Summary

Output to chat:

Skill Functional Audit — 2026-04-14T03:00:00Z
==============================================
Skills audited: 91
  WORKS:   86
  PARTIAL:  2  [tracked in Linear]
  FACADE:   1  [ticket filed]
  STUB:     0
  BROKEN:   0

Audit artifact: ${ONEX_STATE_DIR}/skill-audits/20260414T030000Z.yaml

Gate: PASS (all facades have Linear tickets)

Known Tracked Findings (do not re-flag)

Skills with a currently-open PARTIAL/FACADE verdict and a tracking Linear ticket are listed here so the audit does not re-flag them as new findings. A row stays here only while its ticket is open; once the ticket is Done and the backing node exists, remove the row.

2026-07-01 (, M2c disposition of the 5 no-node facades from ):

• onex:checkpoint — verdict corrected from FACADE to WORKS (misclassified). Real backing exists (omnibase_infra/node_checkpoint_effect, omnibase_infra/node_checkpoint_validate_compute, checkpoint_manager.py CLI) — the audit's naive node_<skill_name> path match doesn't resolve the real node names. SKILL.md now declares the real backing explicitly. Tracked cleanup to repoint the registry / retire the dead omniclaude Polly-shell: .
• onex:dispatch_watchdog — FACADE, de-listed from event-bus dispatch (no deterministic backing anywhere; the omniclaude node is a maturity: stub Polly-shell). Prefer onex:agent_healthcheck (real node-backed via node_worker_stall_recovery). Tracked: .
• onex:epic_team — FACADE, de-listed from event-bus dispatch by design (spawns live session subagents; run.sh shells out to claude -p, still an LLM session, not a node). Tracked: .
• onex:executing_plans — FACADE, de-listed from event-bus dispatch by design (session-native plan review + routing decision). Tracked: .
• onex:wave_scheduler — FACADE, de-listed from event-bus dispatch by design (session-native parallel dispatch; DAG-computation core is a future COMPUTE-node candidate). Tracked: .

The three 2026-04-14 findings have all been resolved and removed from this table:

• onex:pipeline_fill (was FACADE — Done; node_pipeline_fill now exists in omnimarket, so the facade is resolved).
• onex:session (was PARTIAL — Done).
• onex:delegate (was PARTIAL — Done; local-runtime fallback landed).

Update this section when tickets are resolved or new findings are added.

Nightly Scheduling

This skill is scheduled via CronCreate to run nightly:

CronCreate("0 3 * * *", "/onex:skill_functional_audit", recurring=true)

If the audit FAILS (untracked facades found), it writes a friction event to ${ONEX_STATE_DIR}/friction/ and sends a message to the overnight overseer.

Integration with skill-catalog-gap-sweep

The existing skill-catalog-gap-sweep (if it exists) should be updated to call this skill as its functional verification step. File existence alone is no longer sufficient. See knowledge-base-internal:reference/skill-audit-methodology.md.

Related

• Audit report: docs/briefs/skill-functional-audit-2026-04-14.md
• Process doc: knowledge-base-internal:reference/skill-audit-methodology.md
• Prior gap: docs/briefs/skill-catalog-2026-04-14.md (false-clean sweep)
• Tickets: Filed 2026-04-14 for session Phase 2+3, pipeline_fill, and delegate (all resolved)