Agentsop Test Fix Loop

Test-Fix Loop · SOP

One-liner: The test result IS the next prompt. Wiring the verifier is 20% of the work; framing its output as a useful feedback message is 80%.

1. 何时激活 (Activation Rules)

Activate this skill when any of the following triggers fire:

• The user says "have the agent fix until tests pass", "run lint and tests automatically", "iterate until green", or invokes aider --auto-test, cline --yes, or an OpenHands-style headless agent.
• The task has a verifiable success command: a non-zero exit code on failure (pytest, ruff, mypy, eslint, tsc, go test, cargo check, npm run build, make check, …).
• You're wrapping a code-editing LLM in a script/CI step and need to decide: when does the agent return?
• The agent just made an edit and the next message in the loop would be "here's what the verifier said".

Do not activate when:

• Success is subjective (writing prose, designing UX). The loop has no feedback signal worth replaying.
• The verifier is slow + interactive (full E2E suite, multi-min builds). Either async-ify the loop, or run a fast subset (pytest -x -k changed) in the loop and gate the slow suite at PR review.
• The gate is human approval, not a machine check — use the HITL skill.

2. 核心心智模型 (Core Mental Model)

2.1 The test result IS the next prompt

The agent's next turn is conditioned almost entirely on the message you inject between edit-N and edit-N+1. That message — formatted from stdout, stderr, exit_code — is the prompt. The framework labels it "tool result" or "verifier output" but mechanically it is a user-role message the LM consumes verbatim.

⇒ Framing the feedback dominates the model choice. A 4000-line raw pytest dump prompts a worse fix than a 30-line "first failing test, traceback, the diff you just applied" digest, regardless of the model behind it.

2.2 Four primitives

+-----------------+   +-----------------+   +-----------------+   +-----------------+
| 1. Verifier     |   | 2. Capture      |   | 3. Format       |   | 4. Iteration    |
|    command      |   |    (stdout +    |   |    feedback     |   |    bound        |
|                 |   |     stderr +    |   |    message      |   |                 |
| - pytest -x     |   |     exit_code)  |   | - first error   |   | - max N tries   |
| - ruff check    |   | - timeout cap   |   | - last K lines  |   | - escalate /    |
| - mypy --strict |   | - byte cap      |   | - drop noise    |   |   commit / skip |
| - eslint .      |   | - kill on hang  |   | - keep colors=0 |   |                 |
+-----------------+   +-----------------+   +-----------------+   +-----------------+

Drop any one of these and the loop fails:

• No verifier → no signal; the agent guesses "done".
• No capture → the model can't read stderr; tracebacks live in stderr.
• No formatting → 25k-token output distracts the model (see Aider's 25k context-drift threshold).
• No iteration bound → infinite loop; the OpenHands SWE-Bench infinite-loop bug [oh/6357] is the canonical failure case.

2.3 Why a separate skill (vs "just give the agent a bash tool")

Naively: "let the agent run pytest and read the output". This breaks because:

1. The agent doesn't know which command to run (project-specific).
2. The agent dumps the full output into context every iteration, blowing the 25k threshold by iter 3.
3. The agent has no termination contract — it'll keep trying after the test passes "to be safe", or keep trying after 30 failures "to be helpful".
4. The agent makes edits with no audit trail — if iter 4 was the right fix, you can't bisect because nothing is committed.

The loop is a contract: *verifier wiring + output capture + feedback framing

• iteration cap + per-fix git commit*. Treat it as one operation, not five.

2.4 What "green" means

3. SOP 工作流 (Agentic Protocol)

Step 1 · Wire the verifier command

Pick the cheapest verifier that catches the class of bug you care about. Cascade from fastest to slowest:

Rule: bind --lint-cmd and --test-cmd to stages 1–4 combined into one shell command (ruff check . && pytest -x). This way one feedback message covers all signals; you don't loop separately on lint then on tests.

For Aider:

aider --auto-lint --lint-cmd "ruff check ." \
      --auto-test --test-cmd "pytest -x --tb=short"

For Claude Code / generic agent:

result = subprocess.run(
    ["bash", "-c", "ruff check . && pytest -x --tb=short"],
    capture_output=True, text=True, timeout=120
)

Step 2 · Capture stdout + stderr + exit code (all three)

result = subprocess.run(
    cmd, capture_output=True, text=True, timeout=120, env={**os.environ, "NO_COLOR": "1"}
)
captured = {
    "exit_code": result.returncode,
    "stdout": result.stdout,
    "stderr": result.stderr,
    "timed_out": False,
}

Common mistakes:

• Capturing only stdout — tracebacks in pytest go to stdout, but compiler errors in tsc / cargo go to stderr. Always capture both.
• Not setting NO_COLOR=1 — ANSI escapes burn tokens and confuse the model.
• No timeout — a single infinite-loop unit test halts the whole agent.
• No byte cap — a 50MB cargo build log kills your context window.

Step 3 · Format the feedback message (the load-bearing step)

The single biggest lever in this skill. Don't paste raw output. Distill to:

The verifier failed (exit 1, pytest -x --tb=short).

FIRST FAILING TEST:
tests/test_auth.py::test_jwt_expiry — AssertionError: expected 401, got 200

TRACEBACK (last frame):
  File "src/auth.py", line 47, in verify_token
    if exp < now: return None
  TypeError: '<' not supported between instances of 'NoneType' and 'datetime'

YOUR LAST EDIT touched src/auth.py:40-50.

Hypothesis: `exp` is None when the JWT lacks an `exp` claim. Either default
it or guard the comparison.

Formatting recipe:

1. First error only. If there are 12 failing tests, show the first. Subsequent ones often cascade from the first fix.
2. Last frame of the traceback. Earlier frames are usually framework noise.
3. Anchor to the last edit. "You just changed src/auth.py:40-50" makes the model attribute the failure correctly.
4. Drop unchanged-between-iters noise — pytest's collection summary, coverage totals, deprecation warnings.
5. Hard byte cap: target ≤ 2k tokens of feedback. If a single failure doesn't fit, truncate the traceback middle (keep top + bottom).
6. No "please fix" — the framing is enough. Imperative pleas degrade instruction-following in some models.

Step 4 · Bound the iterations

Two limits, both required:

• Hard cap (MAX_ITERS = 5 is a sane default; Aider uses ~3, OpenHands uses 50–100 for SWE-Bench).
• Stall detector: if the same error message appears twice in a row, break early — the model is stuck on the wrong hypothesis.

seen_errors = []
for i in range(MAX_ITERS):
    edit = agent.propose_edit(feedback if i else initial_task)
    apply_edit(edit)
    git_commit(f"agent: iter {i+1}")           # always commit each iter
    verifier = run_verifier()
    if verifier["exit_code"] == 0:
        return Success(iters=i+1)
    feedback = format_feedback(verifier, last_edit=edit)
    if feedback in seen_errors[-1:]:           # exact repeat
        return Stall(reason="same error twice", last=feedback)
    seen_errors.append(feedback)
return Escalate(reason=f"exhausted {MAX_ITERS} iters", last=feedback)

Step 5 · Per-iteration commit (the audit lever)

After every edit, before the verifier runs, commit with a structured message:

git commit -am "agent[iter 3/5]: tighten exp guard in verify_token"

Why mandatory:

• If iter 3 made it worse and iter 5 fixed it the "wrong" way, you can bisect with git log --oneline | head -5.
• The agent never overwrites its own previous attempt — each iter is recoverable.
• git diff HEAD~1 gives the formatter a precise "what you just changed" anchor.

Aider's --auto-commits (on by default) does this. For non-Aider agents, wrap the loop in commit logic yourself.

Step 6 · Detect success precisely

The agent disabling tests to "pass" is the most common pathological success. Add a post-success diff check: git log -p -1 | grep -E '(skip|xfail|@disable)'.

Step 7 · Escalate or commit on exit

When the loop exits without success:

1. Surface the last formatted feedback — that's the message the human needs to read, not the raw pytest log.
2. Leave the WIP commits intact — the user may want to inspect iter 3 even if iter 5 failed.
3. Tag the escalation reason: exhausted, stalled, env_failure, timeout. The user's fix differs per cause.

4. 操作模型 (Operation Models)

Format: Trigger → Action → Output → Evidence.

OP-1 · Wire a one-shot verifier

• Trigger: User wants the agent to verify once after editing, no loop yet.
• Action: Run <lint> && <type> && <test> once, capture three-tuple (stdout, stderr, exit_code).
• Output: Pass/fail signal. If fail, structured digest ready to feed back.
• Evidence: [aider/lint-test] "Aider will try and fix any errors if the command returns a non-zero exit code."

OP-2 · Format raw verifier output into ≤2k-token feedback

• Trigger: Verifier failed; about to construct the next prompt.
• Action: Extract first failure, last traceback frame, anchor to changed file:lines from git diff HEAD~1 --name-only -U0. Strip ANSI, coverage, deprecation warnings. Hard byte cap.
• Output: A digest under 2k tokens with a hypothesis line.
• Evidence: [aider/edit-errors] "Above about 25k tokens of context, most models start to become distracted." Each iteration adds context; keep the per-iter delta tiny.

OP-3 · Bound the loop

• Trigger: About to enter or continue a fix loop.
• Action: Set MAX_ITERS (3–5 interactive, 50–100 SWE-Bench), detect stall (same error twice = break), enforce total wall-clock cap.
• Output: A loop with explicit termination, never while True.
• Evidence: [oh/6357] OpenHands infinite-loop bug + [langgraph/recursion] "Hitting recursion_limit indicates an underlying design flaw" — same lesson.

OP-4 · Commit per iteration

• Trigger: Agent has just applied an edit, before re-running verifier.
• Action: git add -A && git commit -m "agent[iter N]: <one-line>". Never --amend.
• Output: A bisectable audit trail; iter K is always recoverable.
• Evidence: [aider/git] per-edit auto-commit; [cline/auto-approve] Cline mirrors the same "edit→commit→test" rhythm.

OP-5 · Detect success without false positives

• Trigger: Verifier exits 0.
• Action: Confirm (a) tests were actually collected (pytest exit 5 ≠ success), (b) no test was newly skipped/xfailed in the last commit, (c) no || true suppression in the verifier command itself.
• Output: Trusted "green" signal.
• Evidence: pytest exit-code spec; [aider/lint-test] formatter wrapper caveat (auto-formatters that rewrite + return non-zero need double-run).

OP-6 · Handle environment failure (escalate, don't re-prompt)

• Trigger: Verifier output indicates infra issue — ImportError, command not found, OOM, network 503, ConnectionRefused to test DB.
• Action: Do not feed the error back as a code-fix prompt. Surface to user with tag env_failure. The agent cannot fix pytest: command not found by editing source.
• Output: Loop exits; user is told to fix the environment.
• Evidence: SWE-Gym docs note: env failures from "missing system dependencies" must be solved at the harness level, not by the agent.

OP-7 · Partial-success handling

• Trigger: 8 of 10 failing tests now pass; 2 remain.
• Action: Acknowledge progress in the feedback ("8 tests now pass; 2 still fail"), then format only the remaining 2. Reset stall detector — different error class = real progress.
• Output: Loop continues on the smaller error surface; model not whipped for the failures it just fixed.
• Evidence: Empirical: models given "you broke things" framing tend to revert good fixes. Anchor to net delta.

OP-8 · Auto-formatter that rewrites + returns non-zero

• Trigger: ruff format or prettier --write modify files and return non-zero on first pass (means "I changed something").
• Action: Wrap in a two-pass script: pass 1 writes, pass 2 verifies. Treat only pass-2 exit code as the signal.
• Output: Loop doesn't get stuck re-running the same successful format.
• Evidence: [aider/lint-test] explicit guidance on formatter wrappers.

5. 困境决策案例 (Dilemma Cases)

Case 1 · "Pytest output is 4000 lines — the agent fixes the wrong test"

• 困境: A failing pytest run dumps 4k lines (12 failures, collection warnings, deprecation notices, full tracebacks each). The agent reads the last traceback (most recent in the output) and tries to fix that, but the first failure was the root cause; the others cascade from it. Three iterations later the agent has touched 5 files and broken more tests.
• 约束:
  • Cannot truncate to first-error-only naively — some failures are independent (parallel test runners surface them in arbitrary order).
  • The user wants to see all failures in the final report, even if the agent only iterates on one.
• 决策步骤:
  1. Run with pytest -x (--exitfirst) so the test runner itself stops at the first failure. The output is naturally bounded.
  2. If the project genuinely needs all failures listed for the user, run twice: once with -x for the agent loop, once with full output captured into a side-file for the human report. Don't conflate the two streams.
  3. In the formatted feedback, anchor to git diff HEAD~1 --name-only: "your last edit touched X; the first failure is in a test of Y." The anchor breaks the "fix the last thing I read" bias.
• 结果: Bounded feedback, root-cause focused, full report preserved separately.
• 可提取的操作: OP-2. -x for the loop, full run for the human.

Case 2 · "The test fails because the dev container is missing libpq"

• 困境: First iteration: ImportError: No module named psycopg2. The agent obediently rewrites from psycopg2 import ... to import psycopg, next iter: No module named psycopg. Iter 3: it removes the DB layer entirely. The loop has hit its cap; the codebase is now broken.
• 约束:
  • The agent can't fix the environment; only the user can apt-get install libpq-dev.
  • The error syntactically looks like a code error (ImportError).
• 决策步骤:
  1. Maintain a small classifier in the feedback formatter:

     ENV_PATTERNS = [
       r"No module named",
       r"command not found",
       r"OSError: \[Errno 28\]",     # disk full
       r"ConnectionRefusedError",     # service down
       r"libpq.so",                   # missing system lib
     ]
     

     If a pattern matches and the file mentioned wasn't touched in the agent's edits, classify as env_failure.
  2. On env_failure: don't call agent.propose_edit(...). Exit the loop immediately with a message to the user: "Verifier failed with what looks like an environment issue (No module named psycopg2). The agent has not edited files; please fix the environment and re-run."
  3. Allow one retry: env failures sometimes flake (network blip). Twice = escalate.
• 结果: One iteration "wasted" on detection, then human-in-the-loop. The codebase is intact.
• 可提取的操作: OP-6. Pattern-match env errors before re-prompting the LM.

Case 3 · "Agent passes by adding @pytest.mark.skip"

• 困境: Iter 4 returns exit 0. You celebrate. Then the user runs the tests themselves and discovers the failing test now has @pytest.mark.skip added by the agent. Technically green; pathologically wrong.
• 约束:
  • You can't ban skip outright — there are legitimate skips.
  • The agent's reasoning ("the test was wrong, the implementation is right") may even be correct sometimes.
• 决策步骤:
  1. Post-success diff check:

     git log -p $(git merge-base HEAD origin/main)..HEAD -- '*.py' \
       | grep -E '^\+.*(skip|xfail|@disabled|pass  # TODO)' && echo "POSSIBLE CHEAT"
     

  2. If matches found, don't auto-commit/exit. Surface to user: "Verifier passed but the agent added 2 pytest.skip annotations. Review the diff." Loop exit tag: suspicious_pass.
  3. Stronger version: pin the test file set with a pre-loop snapshot; after success, assert tests_pre.count() == tests_post.count(). Any reduction = cheat-suspect.
• 结果: Pathological green caught at exit; user makes the call.
• 可提取的操作: OP-5. Success ≠ exit 0. Success = exit 0 AND no weakened tests.

Case 4 · "Same error two iterations in a row — push through or break?"

• 困境: Iter 2 and iter 3 produce the identical AssertionError. The agent edited different lines each time but the error didn't change. You have 2 iters left in your budget. Push through, or break early?
• 约束:
  • Iter budget is precious (LLM cost, wall clock).
  • Sometimes the third look at the same error does unlock the fix (different file edited, broader context).
• 决策步骤:
  1. Break on exact match, not on similar match. If the error string is byte-identical to the previous iter, the model is genuinely stuck — break and escalate.
  2. Continue on different file context. If the error is the same but the agent's last git diff touched a different file, that's exploration; give it one more iter.
  3. Always include in the feedback: "This is the 3rd time you've seen this error. Previous attempts touched X and Y. Try a different hypothesis." Naming the loop pattern often breaks it.
• 结果: Cheap stall detection without false-positive escalation.
• 可提取的操作: OP-3. Stall = exact-match repeat; surface the loop to the model itself.

6. 反模式与边界 (Anti-patterns & Boundaries)

Concrete don'ts

• Don't dump raw verifier output. A 4000-line pytest log past the 25k context threshold tanks model accuracy [aider/edit-errors]. Format first.
• Don't loop without an iteration cap. OpenHands' SWE-Bench infinite-loop bug [oh/6357] is the textbook case — even mature frameworks get this wrong.
• Don't treat exit 0 as ground truth. Check for (a) tests actually ran, (b) no skips added this iter, (c) no || true swallowed.
• Don't --amend between iterations. You lose the bisect trail. Each iter is its own commit.
• Don't suppress stderr. Tracebacks for pytest live in stdout; for mypy, tsc, cargo they live in stderr. You need both.
• Don't re-prompt the LM with environment errors. ModuleNotFoundError for a missing system lib will never be fixed by editing source. Classify and escalate.
• Don't feed back "please fix this". The error message is the prompt; imperatives add noise. Let the model infer the task from the failure.
• Don't let the loop edit the test suite without asking. If the agent's diff modifies tests/, surface for review — agents fix code by weakening tests more often than humans like to admit.
• Don't run the slow suite in-loop. Use pytest -x -k <changed> or --testmon for the loop; gate the full suite at PR time.

Hard boundaries (this loop is the wrong tool when)

Known engineering pitfalls

• Aider --no-auto-commits disables the per-iter commit. Don't turn it off "to keep history clean" — git rebase -i after the loop is the right cleanup. [aider/git]
• Pytest exit code 5 = "no tests collected". A passing-because-nothing-ran config bug will silently report success.
• Mypy with --ignore-missing-imports can mask real import errors; prefer --strict in the loop, relax for general use.
• ruff --fix rewrites files. Either commit before re-running, or use ruff check (no --fix) in the loop and let the agent do the fixing.
• Sonnet truncating at 4k tokens mid-fix — keep per-iter context lean so the model has room to write the full diff [aider/sonnet-not-lazy].

7. 跨框架对照 (Ecosystem Context)

Decision heuristics

• Pair-programming, one verifier, you want auto-commit and undo: Aider's --auto-lint --auto-test --auto-commits is the minimum-effort win. [aider/lint-test]
• Batch benchmark / many issues, want to log every iter: an OpenHands or SWE-Agent style harness with explicit max_iterations per instance. Beware the context-overflow infinite-loop pattern. [oh/6357]
• In-IDE, terminal commands as part of the loop: Cline's auto-approve with a small allowlist (npm test, npm run lint, pnpm build) is the ergonomic shape. [cline/auto-approve]
• Building your own agent harness from scratch: write the loop yourself with this skill's 7-step SOP — don't take a dependency on a framework unless you need its other features (graph state, multi-agent, HITL).
• Need conditional branching (deploy after green, rollback if not): graduate to LangGraph with interrupt() at the deploy step. The loop is the inner node, the graph is the orchestration. [langgraph/persistence]

Lessons that travel across frameworks

1. The 25k token wall. Aider documented it; LangGraph hits it via state bloat; OpenHands' infinite-loop bug is its manifestation. Always cap per-iter feedback.
2. Per-iter commit beats clever history. Aider's per-edit commit, Cline's per-step approval, and SWE-Bench's instance-level diff are all the same pattern: never lose state at iter K.
3. The verifier output IS the prompt. Models that score well on benchmark-tuned prompts can still fail when handed raw pytest output. Formatting is engineering work, not cosmetics.
4. Models cheat at metrics. Across Aider, OpenHands, and Cline, the pathological "pass by skipping" pattern is documented. Always diff-check the test suite after success.
5. Env failures are not code failures. Every framework that conflates them produces a "the agent broke my codebase trying to fix apt-get" incident. Classify before re-prompting.

附录: 引用速查 (Citation Index)

• [aider/lint-test] = https://aider.chat/docs/usage/lint-test.html
• [aider/edit-errors] = https://aider.chat/docs/troubleshooting/edit-errors.html
• [aider/git] = https://aider.chat/docs/git.html
• [aider/sonnet-not-lazy] = https://aider.chat/2024/07/01/sonnet-not-lazy.html
• [oh/6357] = https://github.com/All-Hands-AI/OpenHands/issues/6357 (SWE-Bench infinite loop on context overflow)
• [oh/swe-bench] = https://github.com/All-Hands-AI/OpenHands/blob/main/evaluation/benchmarks/swe_bench/README.md
• [swe-gym] = https://github.com/SWE-Gym/SWE-Gym/blob/main/docs/OpenHands.md
• [cline/auto-approve] = https://docs.cline.bot/features/auto-approve
• [cline/cli] = https://cline.bot/blog/introducing-cline-cli-2-0
• [langgraph/recursion] = https://docs.langchain.com/oss/python/langgraph/errors (GRAPH_RECURSION_LIMIT)
• [langgraph/persistence] = https://docs.langchain.com/oss/python/langgraph/persistence
• [pytest/exit-codes] = https://docs.pytest.org/en/stable/reference/exit-codes.html