Harness

Agent Harness Coding

The "harness" is the framework that drives the LLM: the Plan-Act state machine, the two role agents, the BaseAgent tool loop, and the event stream. All of it lives in backend/app/domain/services/ and depends only on the Protocols in backend/app/domain/external/ — which is what makes it fully testable offline.

File map (reading order)

Invariants (do not break)

State machine (PlanActFlow):

• Transitions are IDLE → PLANNING → EXECUTING ⇄ UPDATING → SUMMARIZING → COMPLETED. Executor runs one step at a time.
• Successful steps are marked locally and emit PlanEvent(UPDATED) without a Planner round-trip; only step_needs_replan() (step FAILED, or COMPLETED with success=False) enters UPDATING.
• A WaitEvent (from message_ask_user) aborts run() without a DoneEvent; is_done() stays False. Resume path: SessionStatus.WAITING → executor.resume_step, and the user's reply is injected as the tool response to the pending message_ask_user call (see BaseAgent.roll_back).
• On a non-PENDING session, both agents roll_back before the loop starts.
• An empty plan (no steps) completes directly; a single successful step skips summarize (can_skip_summarize).

Tool loop (BaseAgent):

• Every tool_call in an assistant message must receive a tool response — including unknown tools and failed OutputTool validations — so the memory's call/response pairing stays valid for the next LLM request.
• Structured output goes through native function calling (OutputTool); validation errors are fed back as the tool response so the model self-repairs. A plain text reply while an output tool is active gets nudged to call it.
• Context budgets: tool results are truncated at ingestion (max_tool_result_chars, 16k chars); memory is compacted before LLM calls when over max_context_tokens (100k). Compaction elides old tool results but preserves the message skeleton.
• invoke_tool retries max_retries (3) times; the loop caps at max_iterations (100).
• Planner has tool_choice="required" and holds no executor toolkits — only a compact capability overview (describe_toolkits), keeping full schemas out of its context.
• Executor rejects complete_step(success=true) unless real work tools ran (_WORK_TOOLKITS).
• StructuredOutputEvent is internal — never part of the public event union streamed to clients.
• Planner and Executor keep separate memories, keyed agent_id:name in AgentRepository — this is what makes sessions resumable.

Extension recipes

Add a toolkit: subclass BaseToolkit in tools/, decorate methods with @tool (docstring Args become the schema), optionally set instructions (becomes a <name_rules> system-prompt section). Wire it into the tools list in flows/plan_act.py. For rich UI rendering add a tool content type in ../models/event.py and a renderer in frontend/src/components/toolViews/.

Add an event type: define it in ../models/event.py, yield it from the flow/agent, handle it in the frontend WebSocket client (frontend/src/api/). Events reach the browser via Redis message queues → /api/v1/ws/chat.

Change prompts: edit prompts/; test_plan_act_prompts.py guards against stale tool references. System prompts are assembled by prompts/system.py:build_system_prompt (core + toolkit sections + role + project instructions).

Change external capabilities: define the Protocol in domain/external/ first, implement in infrastructure/external/, wire in interfaces/dependencies.py. The harness must keep depending on the Protocol only.

Testing pyramid

1. Offline unit tests (seconds — always run these when touching the harness):

cd backend && uv run pytest tests/test_plan_act_flow.py \
  tests/test_context_engineering.py tests/test_single_loop_manus.py -q

Shared fakes live in backend/tests/harness.py: ScriptedLLM (scripted assistant replies, records every request), FakeAgentRepository, FakeSandbox, FakeSession(Repository), StubAgent, plus build_plan_act_flow / build_agent_loop_flow and the create_plan_call factory. Import from there — do not redefine fakes per test module. A scripted test is: list the exact LLMMessage.assistant(tool_calls=[...]) turns, run the flow, assert on the yielded events and on llm.asked_tool_names / llm.requests; end with assert llm.responses == [].

Gotchas: run uv sync first; Settings reads real env vars, so unset API_BASE etc. when testing config defaults (env -u API_BASE uv run pytest …).

2. Behavioral evals (offline, deterministic — backend/evals/):

cd backend && uv run python -m evals.run            # exit 1 on failure
cd backend && uv run python -m evals.run --json out.json --scenario ask_user_waits

Each scenario in evals/scenarios.py drives the real PlanActFlow with a scripted LLM and scores the run (evals/metrics.py): completion, LLM-call budget, replans, self-repairs, premature-complete rejections, error events. Add a scenario when you add or change a harness behavior; the report is a regression gate for prompt/loop changes.

3. API e2e over the real stack (cd backend && uv run pytest -m e2e): with the dev stack up (./dev.sh up -d), tests/test_e2e_plan_act.py creates a session over the real API, drives the chat WebSocket, lets the mockserver replay a scripted scenario and the real sandbox execute tools, then asserts on the wire events (plan/step/tool/message/status_update/stream_end). Tests self-skip when the stack is down. Switch scripts programmatically via the mockserver control API (no restart needed):

curl -X POST localhost:8090/mock/scenario -H 'Content-Type: application/json' \
  -d '{"file": "plan_act_e2e.yaml"}'   # also resets the replay index
curl localhost:8090/mock/scenario      # inspect file/index
curl -X POST localhost:8090/mock/reset # back to MOCK_DATA_FILE default

Mockserver replays a YAML list of chat completions in order (default file via MOCK_DATA_FILE, per-reply delay via MOCK_DELAY). Wire-format note: the WS plan event carries only flat steps (no title/status); titles arrive as title events, and step events are flat (id/status/result).

Script-writing gotcha: a step that only asked the user cannot complete_step(success=true) — the executor rejects it until a work tool (shell/file/browser/search/mcp) ran in that step. Script real work before completing.

4. Browser e2e (cd frontend && npm run test:e2e): Playwright (frontend/e2e/plan-act.spec.ts, config frontend/playwright.config.ts) drives the real UI at localhost:5173 as a user: type a task into the TipTap chat box, press Enter, and assert the rendered chat timeline (plan message, step row, final answer, "Task completed" badge, waiting banner + resume). Needs the dev stack up and browsers installed (npx playwright install chromium). Runs serially (workers: 1) because the mockserver replay index is global. UI assertion gotcha: the plan title from TitleEvent goes to the sidebar/tab — assert on chat-visible texts (plan message, step description, deliver_result message) instead.

5. Full stack manual: ./dev.sh up -d, open http://localhost:5173, watch ./dev.sh logs -f backend. Needed for sandbox/browser/VNC behavior that automation can't cover.

Maintenance

When you change flow transitions, event contracts, memory/rollback semantics, or output tools, update the Invariants section above, the affected scripted tests, and the eval scenarios in backend/evals/ in the same PR.