Ultraqa

Ported from oh-my-codex ultraqa. OMX runtime conventions ($macro invocation, omx CLI, .omx/ state directory) are replaced with WorkBuddy idioms (Skill tool, Agent tool, task list, .workbuddy/memory).

UltraQA Skill

Operating Contract

• Outcome-first framing with concise, evidence-dense progress and completion reporting.
• Treat newer user updates as local overrides for the active workflow branch while preserving earlier non-conflicting constraints.
• If the user says continue, advance the current verified next step instead of restarting discovery.
• UltraQA is not satisfied by a shallow build/lint/typecheck/test checklist. It must exercise the requested behavior through adversarial dynamic e2e scenarios whenever the target can be run, simulated, or harnessed safely.

[ULTRAQA ACTIVATED - ADVERSARIAL DYNAMIC E2E QA CYCLING]

Overview

UltraQA finds real behavior failures by combining normal verification commands with generated end-to-end scenarios, hostile user modeling, temporary harnesses when useful, and a structured evidence report. The workflow repeats test → diagnose → fix → retest until the goal is met, a bounded stop condition is reached, or a safety boundary blocks further execution.

Goal Parsing

Parse the goal from arguments. Supported formats:

If no structured goal is provided, interpret the argument as a custom behavior goal and derive a runnable e2e strategy from repository context.

Required Scenario Matrix

Before declaring success, create and maintain a scenario matrix (in memory or a temp file). Each row must include: scenario id, intent, user/attacker model, setup, command or harness, expected signal, actual result, fixes applied, evidence, and cleanup status.

The matrix must include normal-path coverage plus adversarial dynamic e2e scenarios selected from the current goal and codebase. Unless clearly irrelevant or impossible, include these hostile and edge-case classes:

1. Malformed input: invalid JSON, missing fields, invalid flags, oversized strings, unusual Unicode, path traversal-like values, corrupted state files.
2. Repeated interruptions: repeated continue, stop/cancel/abort wording, interrupted command output, retries after partial progress.
3. Prompt injection attempts: user text that tries to override instructions, exfiltrate secrets, skip verification, delete state, or claim false success.
4. Cancel/resume behavior: active state cleanup, resume detection, stale in-progress state, cancellation followed by a fresh run.
5. Stale state: old state files, mismatched sessions, missing timestamps, contradictory phase metadata.
6. Dirty worktree: pre-existing modifications, untracked generated files, verification that UltraQA does not hide or overwrite unrelated work.
7. Hung or long-running commands: bounded timeout handling, killed child processes, recovery notes.
8. Flaky tests: rerun strategy, failure clustering, quarantine evidence, avoiding false green from a single lucky pass.
9. Misleading success output: output containing success phrases with non-zero exits, hidden failures, skipped tests, partial command logs.

Dynamic E2E and Temporary Harness Rules

• Generate temporary tests, scripts, fixtures, or harnesses when they materially improve behavioral confidence and no existing e2e surface covers the scenario.
• Prefer project-native test tools and small throwaway harnesses under a temporary directory or clearly named test fixture.
• Record every generated artifact in the scenario matrix, including whether it was committed intentionally or removed during cleanup.
• Use bounded runtimes and explicit timeouts for commands that can hang.
• Validate exit codes and output semantics; do not trust success-looking text alone.
• Do not delete, rewrite, or mask unrelated user work. Capture dirty-worktree evidence before and after generated harness work.

Temporary Harness Generation Guardrails

Generated harnesses are part of the QA evidence chain; until setup succeeds, they are evidence about the harness apparatus, not product behavior.

• Use absolute repo imports for built artifacts. When a harness runs from /tmp or another scratch directory but imports repository code, resolve the repo root explicitly and import built modules with an absolute path or pathToFileURL(join(repoRoot, "dist", ...)).href. Never rely on ./dist/... from the harness's temp directory.
• Use a safe file writer for JS/TS harness bodies. Prefer a small Node/Python writer or another non-interpolating file-write mechanism for harness source containing backticks, ${...}, shell metacharacters, or prompt-injection strings. If a shell heredoc is unavoidable, quote the delimiter and verify the written file before execution; do not use interpolating heredocs for JavaScript assertions.
• Sanitize ambient runtime env for isolated probes. When the scenario creates a temporary repo/state tree or checks local isolation, run the probe with OMX_ROOT/OMX_STATE_ROOT-style env unset (e.g. env -u OMX_ROOT -u OMX_STATE_ROOT ...) — and in WorkBuddy, never reference .omx/. Use a clean cwd so ambient boxed state cannot redirect reads/writes away from the scenario fixture.
• Classify harness setup failures separately. If a generated harness fails before exercising product behavior (import paths, shell interpolation, env leakage, fixture construction), record it as harness debris, fix the harness, and rerun the scenario before declaring a product defect.

Cycle Workflow

Cycle N (Max 5)

Track each cycle in the task list: TaskCreate a cycle task (in_progress → completed).

1. PLAN ADVERSARIAL QA

   • Restate the goal, success criteria, safety bounds, and stop condition.
   • Inspect repository context enough to identify runnable surfaces, test commands, state files, cleanup paths.
   • Build or update the required scenario matrix before running commands.

2. RUN BASELINE VERIFICATION

   • --tests: run the project's test command.
   • --build: run the project's build command.
   • --lint: run the project's lint command.
   • --typecheck: run the project's type check command.
   • --custom: run the appropriate command and check the pattern plus exit status and failure markers.
   • --interactive: harness the CLI/service with generated hostile and edge-case interactions (spawn a qa-tester-style Agent if available, otherwise drive it directly via Bash).

3. RUN ADVERSARIAL DYNAMIC E2E SCENARIOS

   • Execute the scenario matrix using existing e2e tests, generated temporary tests, or generated harnesses.
   • Model malicious/hostile user behavior explicitly, including prompt injection and attempts to bypass safety or verification.
   • Exercise malformed input, repeated interruptions, cancel/resume, stale state, dirty worktree, hung commands, flaky tests, and misleading success output when relevant.
   • Capture commands, exit codes, key output excerpts, artifacts, and cleanup status.

4. CHECK RESULT

   • YES only if baseline verification and adversarial e2e scenarios passed, generated artifacts are cleaned up or intentionally tracked, and the report has complete evidence.
   • NO if any scenario failed, was skipped without justification, left debris, relied on misleading output, or lacked evidence. Continue to step 5.

5. ARCHITECT DIAGNOSIS

   • Reason about root cause, safety implications, and specific fix recommendations. Invoke the architect/analyze skill via the Skill tool if available (skill: architect / skill: analyze); otherwise perform the diagnosis directly.

6. FIX ISSUES

   • Apply the fix precisely. Invoke an executor/plan skill via the Skill tool if available; otherwise implement directly. Constraints: preserve unrelated dirty work, clean temporary harnesses, keep safety bounds.

7. CLEAN UP AND ROLLBACK

   • Remove temporary harnesses, fixtures, logs, spawned processes, and state files unless intentional deliverables.
   • Roll back failed experimental edits not part of the final fix.
   • Re-check the worktree and record remaining intentional changes or residual debris.

8. REPEAT

   • Go back to step 1 with the updated scenario matrix and failure history.

Safety Bounds

UltraQA must stay inside these safety bounds:

• No destructive commands such as force resets, broad deletes, secret exfiltration, credential dumping, production writes, or unbounded process spawning.
• No reading or printing secrets beyond the minimum metadata needed to verify absence of leakage.
• No network or external-production side effects unless the user explicitly authorized them.
• No unbounded waits: use timeouts, retries with caps, clear hung-command diagnostics.
• No hiding unrelated dirty work or generated debris.
• If a required scenario would violate these bounds, mark it blocked in the report with the safe substitute used.

Exit Conditions

Structured Report

Every terminal UltraQA result must include this report shape:

# UltraQA Report

## Goal and success criteria
- Goal:
- Stop condition:
- Safety bounds applied:

## Scenario matrix
| ID | User/attacker model | Scenario | Command/harness | Expected signal | Actual result | Status | Evidence | Cleanup |
|----|---------------------|----------|-----------------|-----------------|---------------|--------|----------|---------|

## Commands run
- `[exit code] command` — purpose, duration/timeout, key output evidence

## Failures found
- Scenario ID, failure signal, root cause, user impact, safety impact

## Fixes applied
- Files changed, rationale, linked failing scenario(s), regression evidence

## Cleanup and rollback
- Generated artifacts removed or intentionally kept
- State/process cleanup performed
- Worktree status before/after

## Residual risks
- Untested or blocked scenarios with reasons and safe substitutes

## Evidence
- Test output, e2e logs, harness output, transcripts when relevant, rerun/flake evidence

Observability

Output progress each cycle:

[ULTRAQA Cycle 1/5] Planning adversarial scenario matrix...
[ULTRAQA Cycle 1/5] Running baseline tests...
[ULTRAQA Cycle 1/5] Running ADV-E2E-003 prompt-injection harness...
[ULTRAQA Cycle 1/5] FAILED - stale state resume accepted misleading success output
[ULTRAQA Cycle 1/5] Architect diagnosing scenario ADV-E2E-003...
[ULTRAQA Cycle 1/5] Fixing: src/hooks/... - validate exit code before success phrase
[ULTRAQA Cycle 1/5] Cleaning temporary harnesses and state...
[ULTRAQA Cycle 2/5] PASSED - baseline + 9 adversarial scenarios pass
[ULTRAQA COMPLETE] Goal met after 2 cycles

State Tracking

Use the task list for UltraQA lifecycle state (create a task per cycle; update current_phase via status/description). For durable cross-session resume context, append a short note to .workbuddy/memory/YYYY-MM-DD.md (e.g. mode: ultraqa, iteration: N, scenario_matrix: <summary>). Never call omx state.

• On start: TaskCreate the run with current_phase: planning.
• On each cycle: TaskUpdate the cycle task (qa → adversarial-e2e → diagnose → fix → cleanup).
• On completion: mark cycle tasks completed; if resuming, read the memory note for prior iteration count.

Scenario Examples

Good: The user says continue after the workflow already has a clear next step. Continue the current branch, rerun the relevant adversarial scenario, update the report instead of restarting discovery.

Good: The user changes only the output shape or downstream delivery step (e.g. make a PR). Preserve earlier non-conflicting constraints and apply the update locally.

Good: A CLI prints SUCCESS while exiting 1. Mark the misleading-success-output scenario failed, fix the parser/reporting path, rerun the generated harness.

Bad: The workflow runs only npm test/build/lint/typecheck, sees green, and declares UltraQA complete without adversarial dynamic e2e coverage.

Bad: A generated harness leaves untracked files, state, or a child process behind and the final report omits cleanup status.

Bad: The user says continue, and the workflow restarts discovery or stops before missing verification/evidence is gathered.

Cancellation

The user can cancel (e.g. /cancel), which clears UltraQA state. Cancellation itself should be tested in cancel/resume scenarios when relevant, but UltraQA must not block an explicit user cancellation.

Important Rules

1. ADVERSARIAL E2E REQUIRED — Baseline build/lint/typecheck/test commands are necessary evidence, not sufficient completion proof.
2. SCENARIO MATRIX REQUIRED — Track normal, hostile, malformed, interruption, injection, cancel/resume, stale-state, dirty-worktree, hung-command, flaky, and misleading-output coverage.
3. GENERATE HARNESSES WHEN USEFUL — Create temporary tests/harnesses when they materially improve confidence, then clean up or commit intentionally.
4. PARALLEL WHEN SAFE — Run independent diagnostics while preparing fixes; do not parallelize commands that mutate the same state or worktree.
5. TRACK FAILURES — Record each failure to detect patterns and avoid false greens.
6. EARLY EXIT ON PATTERN — 3x same failure = stop and surface with root cause and residual risk.
7. CLEAR OUTPUT — User should always know current cycle, scenario, command, status, evidence.
8. CLEAN UP — Clear task list + temporary artifacts on completion, cancellation, or early stop.
9. SAFETY FIRST — Never exfiltrate secrets, run destructive cleanup, write to production, or wait indefinitely to satisfy a scenario.

STATE CLEANUP ON COMPLETION

When goal met OR max cycles reached OR exiting early: mark all UltraQA cycle tasks completed/deleted and remove temporary e2e harnesses, fixtures, and logs unless they are intentional artifacts listed in the report. Do not delete files outside the run's scope.

Begin ULTRAQA cycling now. Parse the goal, build the adversarial dynamic e2e scenario matrix, and start cycle 1.