Polly E2e Dev

polly orchestrator: end-to-end CUJ dev & testing

polly (examples/polly/) is a multi-agent coding orchestrator: a claude-sdk "brain" that writes no code itself and delegates everything to three coding sub-agents — claude_code (claude-native), codex (codex-native), and pi (headless, multi-model). Its critical user journeys are orchestration behaviors, not single-turn answers:

• roster preflight — first turn runs command -v claude codex pi, routes only to workers whose CLI resolved.
• investigate — read-only work fanned to explore/search sub-agents; synthesize from their reports.
• fanout — independent tasks, each in its own git worktree + sub-agent, each opening its own PR.
• cross-review — an implementer's diff is verified by a different-vendor sub-agent (diff + contract only); blocking issues become fix-tasks.
• plan gate / inbox — pull the human in at the plan gate; supervise via the inbox + autowake, never busy-poll.
• guardrails (omnigent.inner.nessie.policies) — blast_radius (deny force-push / rm -rf /), spawn_bounds (cap dispatches per turn), headless_subagent_purpose_guard (every dispatch needs args.purpose).

This skill tests those CUJs two ways. Use both — they cover different things:

Like the sibling harness skills, turns run from your current checkout (omni run <bundle> --server <url> = local runner + remote server), so testing exercises exactly the code you're on.

Interpreter

The driver and CLI need the repo's Python ≥3.12 env. If .venv/ is missing, create it once from the checkout:

uv run --frozen python -c "import omnigent; print('ok')"   # builds .venv

Then use .venv/bin/python / .venv/bin/omni below.

Part A — the deterministic mock loop (polly_cuj.py)

The driver boots a throwaway local Omnigent server (which carries omnigent.inner.nessie.policies — the module polly's guardrails resolve) plus the repo's mock-LLM server, rewrites the polly bundle to the openai-agents harness wired to the mock, then runs omnigent run turns where the brain is scripted (text or tool calls). It prints one SUMMARY {json} per scenario and exits non-zero if any check failed.

.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --list-scenarios
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --scenario all
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --scenario guardrail_purpose --keep

Read the result with … | grep '^SUMMARY' | python -m json.tool. Each run takes ~45–55s for all five scenarios; no credentials or egress are required.

Scenario catalog

The verifiable before→after loop

The driver exists for a loop, not a one-shot. To prove a fix:

1. On the unfixed code, run the scenario → a check is false (baseline).
2. Make the change.
3. Run the same scenario → the check flips to true.

A fix is "verifiable" only if a check flips. If it doesn't flip, you can't prove the change did anything — keep working. To cover a new mechanism, add a scenario_* function + a row in _SCENARIOS (each builds a bundle, scripts the mock, runs a turn, and asserts an observable effect — a session item, a deny sentinel, a file on disk).

What the mock loop can and can't prove

It tests mechanics because the brain is scripted: tool dispatch, the guardrail gate, session persistence, fan-out plumbing. It does not test polly's judgment (whether the real brain preflights, decomposes, picks the right vendor, cross-reviews). That is the live recipe.

Part B — the live recipe (real claude/codex/pi)

Prereqs (check first)

1. You're on the branch you want to test.
2. A Claude provider for the brain (omni setup, or ANTHROPIC_API_KEY, or a Databricks default). Verify booleans only — never print keys.
3. Worker CLIs on PATH — this is the roster preflight:

   command -v claude codex pi || true
   

   A worker is launchable only if its binary resolved. Cross-review needs two different vendors available.
4. Network egress to the model backends; gh authed if you want real PRs.

Run a live turn

.venv/bin/omni server --background && .venv/bin/omni server status   # prints $SERVER, e.g. http://127.0.0.1:6767
SERVER=http://127.0.0.1:6767
timeout 280 .venv/bin/omni run examples/polly \
  -p "Investigate how the runner enforces tool-call policies and report file:line evidence." \
  --server "$SERVER" 2>&1

Always pass --server "$SERVER"; omitting it routes to the configured remote deploy, which may be stale and reject parts of the bundle.

Observe CUJs (CLI + HTTP API + filesystem)

Grab the session id, then read the transcript and the side effects:

SID=$(curl -s "$SERVER/v1/sessions?kind=default&order=desc&limit=1" | python -c "import sys,json;print(json.load(sys.stdin)['data'][0]['id'])")
curl -s "$SERVER/v1/sessions/$SID/items"          | python -m json.tool | tail -60   # brain transcript + tool calls
curl -s "$SERVER/v1/sessions/$SID/child_sessions" | python -m json.tool             # dispatched sub-agents
git worktree list                                  # fanout: one per task
cat .polly/registry.json 2>/dev/null               # polly's task list
gh pr list --author "@me"                          # each implementer opens its own PR

Per-CUJ live playbook

For the guardrail DENY set (force-push, rm -rf /, unmarked dispatch, fan-out cap), prefer the mock loop — it's deterministic and creates no real side effects.

CUJ coverage map

Known sharp edges (found while building this skill — verify, may change)

• spawn_bounds per-turn cap does not trip in the local server-side path. The cap is a stateful per-turn counter, but the server rebuilds the policy engine per tools/call (_build_policy_engine_from_spec, sessions.py), so the counter resets every call. Stateless policies (purpose_guard, blast_radius) are unaffected. fanout_dispatch reports this as a finding rather than failing. Verify the cap live, where a persistent per-turn engine applies.
• Two deny formats. Bridged sys_* tools surface a denial as {"error": "Denied by policy: <reason>"}; SDK function tools use [Denied by policy: <name>] {json}. Both share the Denied by policy: marker — match on that plus a policy-specific reason fragment (the driver does).
• Live fan-out needs the worker CLIs. In the mock loop, sub-agents are rewritten to openai-agents so a dispatch needs no binary. Live, a missing claude/codex/pi makes that worker fail to boot — treat it as UNAVAILABLE.
• Default server gotcha. config.yaml's server: points at a remote deploy; always pass --server "$SERVER" for local testing.

Code & tests

• Bundle / prompt / guardrails: examples/polly/config.yaml
• Sub-agents: examples/polly/agents/{claude_code,codex,pi}/config.yaml
• Orchestration skills: examples/polly/skills/{investigate,fanout,cross-review}/SKILL.md
• Guardrail policies: omnigent/inner/nessie/policies.py
• Runner-side gate: omnigent/runner/policy.py; server-side tool-call enforcement: omnigent/server/routes/sessions.py
• Mock LLM server: tests/server/integration/mock_llm_server.py

# Existing pytest e2e for polly (mock-LLM) — complementary to this skill:
uv run --frozen --group test python -m pytest \
  tests/e2e/test_polly_e2e.py \
  tests/e2e/test_polly_cost_advisor_e2e.py \
  tests/e2e/test_polly_subagent_model_e2e.py -q

Teardown — non-negotiable

The driver reaps everything it starts, including the per-conversation omnigent.host._daemon_entry / runner._entry / harnesses._runner subprocesses an omni run turn spawns (a plain server SIGTERM leaves these orphaned). The sweep is scoped to this interpreter, so it never touches another worktree. After a live session, sweep manually:

.venv/bin/omni server stop
pgrep -af "$(pwd)/.venv/bin/python -m omnigent" | grep -E "_entry|_runner|_daemon" || echo clean

Honesty

If a worker CLI, credential, or egress isn't available, say the live CUJ was skipped — don't claim it passed. The strongest evidence is a reproduced baseline plus the flipped check (mock loop) or the observed round trip in …/items + …/child_sessions (live). Report the real SUMMARY lines, not a summary of a summary.