# Meta Harness Search R1

> Run one iteration of Search-R1 harness evolution.

- Skill: `krafton-ai/meta-harness-search-r1` (Agent Skill)
- Install (CLI): `npx skillmds@latest add krafton-ai/meta-harness-search-r1`
- Raw SKILL.md: https://api.skillmd.com/api/skills/krafton-ai/meta-harness-search-r1/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: krafton-ai (https://skillmd.com/u/krafton-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/krafton-ai/meta-harness-search-r1

---


# Meta-Harness (Search-R1) — Harness Evolution

Run ONE iteration of harness evolution. Do all work in the main session — do
NOT delegate to subagents. Constraints get lost when you delegate.

**You do NOT run benchmarks.** You analyze results + failed trajectories,
prototype changes, and implement new harnesses. The outer loop
(`meta_harness_search_r1.py`) handles benchmarking separately.

## CRITICAL CONSTRAINTS

- You MUST produce the exact number of candidate `harness.py` files the task
  prompt asks for, one per listed slot.
- Each file MUST define `class CandidateEnv(SearchR1Env)`.
- Each candidate is exactly ONE mechanism — do NOT bundle unrelated ideas.
- Do not stop early or claim the current harness is optimal.

### Anti-parameter-tuning rules

The most common failure mode is creating candidates that are just parameter
variants of existing harnesses. Check `evolution_summary.jsonl` for what's
been tried.

**Good candidates change a fundamental mechanism:**

- A new query-rewriting strategy (e.g. LLM-side decomposition, entity
  extraction, embedder-specific prefixing)
- A new `env_response` structure (e.g. dedup-detection nudge,
  retrieval-feedback injection, format-correction loop)
- A new SYSTEM_PROMPT architecture (e.g. ReAct-style explicit plan,
  chain-of-verification, plan-then-execute)
- A new `search` semantics (e.g. score-based re-ranking, follow-up query
  expansion, retrieval-with-evidence-snippets)

**Bad candidates just tune numbers.** If `search()` / `env_response()` /
SYSTEM_PROMPT structure are byte-identical to the base except for constants
or single-word edits, it's a parameter variant. REWRITE with a truly novel
mechanism.

**Combining harnesses is valid.** Take the search reformulation from one
prior harness and the SYSTEM_PROMPT from another; or draw on published
approaches (ReAct, IRCoT, Self-Ask, Search-o1).

**Exploitation axes:** A=SYSTEM_PROMPT structure, B=USER_PROMPT_TEMPLATE,
C=`search()` query rewriting, D=`search()` retrieval params (topk /
max_doc_tokens), E=`env_response` nudge logic, F=stop-condition / turn
budgeting. If the last 3 iterations explored the same axis, pick different
ones.

### Anti-overfitting rules

- **No question-specific hints.** Do not hardcode answers, entities, named
  subjects, or branching on question text.
- **Never mention dataset or subset names in code.** No
  `if "HotpotQA" in question:` branches, no `data_source`-conditional
  logic, no question-pattern hardcoding in SYSTEM_PROMPT /
  USER_PROMPT_TEMPLATE / `search` / `env_response`. Reading `data_source`
  in your *analysis* (Step 1) to group failures is fine; embedding it in
  candidate code is not.
- **General patterns are OK.** Rules like "always consult the retriever
  before producing a final answer" or "normalize whitespace in queries"
  apply broadly — fine.
- **If in doubt, make it more general.**

## CONTEXT

You are evolving a harness for multi-turn factual QA over wiki-18 (≈21M
Wikipedia passages, e5-base-v2 + FAISS, served over local HTTP — NOT a web
search engine). The eval is English factual QA — see `num_examples` in
`val.json` and `data_source` on each trajectory entry for the actual
composition of the current run.

**Read these files for ground truth** — do NOT duplicate facts here:

- `environments/search_r1/base_harness.py` — baseline `SearchR1Env`, default
  `search()`, default `env_response()`, default SYSTEM/USER prompts. Your
  starting point.
- `environments/search_r1/env.py` — `correct_answer()` reward, dataset
  loading, candidate schema validation. Authoritative for reward semantics.
- `configs/tool_config/search_tool_config.yaml` — tool schema seen by the
  verl GRPO trainer. CandidateEnv must match this schema.

Eval set size varies by run; `num_examples` in `val.json` is authoritative.
`data_source` on each trajectory entry identifies which subset that example
came from.

### SCHEMA CONTRACT (DO NOT BREAK)

Your `search()` tool's first parameter MUST be `query_list: list[str]` with
`maxItems=1`. verl's GRPO loader reads the yaml schema; if `CandidateEnv`
breaks this contract, every tool call during training fails with
`TypeError: unexpected keyword argument 'query_list'`, pinning reward to
~0. `env.py:_validate_candidate_search_signature` enforces this at load
time and raises `ValueError` if violated.

verl SearchTool's lenient policy is mirrored in `base_harness.py:search()`:
when the model emits >1 query, keep the first and silently drop the rest.
Maintain this parity if you redefine `search`.

### Retrieval payload — required fields

The retriever server requires this payload shape; missing fields cause
`KeyError: 'document'` server-side:

```python
{
    "queries": [query],
    "topk": topk,                       # function arg, default 3
    "return_scores": True,
    "max_doc_tokens": max_doc_tokens,   # function arg, default 200
    "tokenizer_name": "Qwen/Qwen3.5-2B",
}
```

Both `topk` and `max_doc_tokens` are exposed as `search()` keyword arguments
(see base `search(query_list, topk=3, max_doc_tokens=200)`), so candidate
harnesses can re-tune them axis-D-style by changing the defaults or by
threading per-turn values through `env_response` — the model can also pass
them in its tool call. Whatever values reach the retriever must still be
forwarded in the payload exactly as shown above; do not hardcode `200` or
`3` again in the payload dict.

### Turn budget — `MAX_TURNS` class attribute

`SearchR1Env` (and therefore every `CandidateEnv`) exposes a class
attribute `MAX_TURNS: int = 4` that controls the multi-turn ceiling
(axis F). Override it the same way you'd override `SYSTEM_PROMPT` to give
the harness more search/answer rounds:

```python
class CandidateEnv(SearchR1Env):
    MAX_TURNS = 8           # = base, default 4
    ...
```

The hard ceiling is **16**. `env.py:load_environment` clamps any value
above 16 down to 16 and logs a warning, and `slurm_train.sh` applies the
same clamp before forwarding the value to verl GRPO as
`actor_rollout_ref.rollout.multi_turn.max_assistant_turns`, so the
training and meta-harness eval rollouts always agree on the cap. Asking
for `MAX_TURNS = 24` is therefore safe — the run still proceeds at 16 —
but you do not gain extra depth past 16, so design within that envelope.

The tool_response returned to the model is
`json.dumps({"result": "<stitched docs>"}, ensure_ascii=False)` — NOT a raw
`"Doc N (Title: ...)..."` string. Match this if you redefine `search`.

## Reward semantics (from `env.py:correct_answer`)

- No `<answer>...</answer>` tag → **0.0**
- Tag present but normalized EM check fails → **0.0**
- Tag present, EM match, but `<answer>` count > 10 (spam penalty) → **0.25**
- Tag present, EM match, normal count → **1.0**

There is NO format-only partial credit. `format_reward` is a **separate
logging metric** (1 if any answer tag exists, else 0) — it does NOT
contribute to the reward number. It shows up as a separate column
(`avg_format_reward`) in `evolution_summary.jsonl` for diagnostic use only.

`success_rate` = `avg_reward` = average of `correct_answer` across rollouts.
Pareto frontier ranks by `(avg_success_rate, mean_turn_count)`.

## KEY FILES TO READ

Before proposing, read (relative to your working directory):

1. `logs/accepted_harness.txt` — current best harness name (e.g. `h0`)
2. `harnesses/<accepted>/harness.py` — current harness code
3. `logs/frontier_val.json` — Pareto frontier (success_rate × mean_turn_count)
4. `logs/evolution_summary.jsonl` — past results: one JSON per evaluated
   candidate with success/correct/format rewards, turn count, pareto flag,
   hypothesis, changes, axis, components
5. Latest `logs/iteration_*/comparison.json` snapshots + `report.md` if present
6. `logs/<profile>/<accepted>/<model>/val.json` — `failure_trajectories[]`
   (up to 5 failed rollouts) and `success_trajectories[]` (up to 2). Each
   entry is `{example_id, reward, turn_count, trace}` where **`trace` is a
   single concatenated string** (`[SYSTEM]...[USER]...[ASSISTANT]...
   [TOOL]...`), not a structured dict. Fast to skim, low-fidelity.
7. `logs/<profile>/<accepted>/<model>/trajectories.jsonl` — one JSON line
   per rollout (ALL 256, not capped). Each line:
   `{example_id, reward, correct, stop_condition, turn_count, question,
   answer, harness_trace, completion}` — `harness_trace` is a list of
   per-turn `{turn, action_type, assistant_action, env_feedback}` dicts;
   `completion` is the raw chat messages. Use this for high-fidelity
   per-turn analysis.

## ANALYSIS — Failure mode discovery

Skim `val.json:failure_trajectories[]` for a quick preview (5
concatenated-string traces). Then open `trajectories.jsonl` and read 5–10
failed rollouts end-to-end using the structured `harness_trace` +
`completion` fields. Describe what mechanism actually broke in each —
do NOT pattern-match to a pre-defined catalog. Note the `data_source` of
each failure for analysis purposes: different subsets often fail for
different reasons. Note also that wiki-18 is a 2018 snapshot, so some
"failures" are corpus gaps, not harness flaws.

## WORKFLOW

### Step 0: Post-eval reports (write if missing)

For each `logs/iteration_*/` that has `comparison.json` but no `report.md`,
write `logs/iteration_<NNN>/report.md` (≤ 30 lines) covering:

- Which candidate(s) became accepted, or why none did
- Which axes (A–F) were explored this iteration; outcome
- One concrete takeaway for the next iteration

These accumulate cross-iteration context so the proposer doesn't re-derive
mechanism understanding from raw `evolution_summary.jsonl` every time.

### Step 1: Analyze

Read the accepted `harness.py`, `frontier_val.json`,
`evolution_summary.jsonl`, latest `val.json` / `trajectories.jsonl`. For
each candidate slot, form ONE falsifiable hypothesis tied to a specific
failure mechanism you observed. K slots → K distinct hypotheses on different
axes.

### Step 2: Prototype — MANDATORY

**You MUST prototype your mechanism before writing the final harness.py.**
Do NOT skip this step. Candidates that skip prototyping tend to have bugs.

For each candidate that involves NEW retrieval logic, NEW `env_response`
logic, or NEW query rewriting:

1. Write a test script in `/tmp/` that exercises the core mechanism in
   isolation. Examples:
   - A custom `search()`: call it on 3–5 real questions from
     `trajectories.jsonl` and print the resulting `<tool_response>`.
   - A new `env_response` nudge: simulate a dummy assistant message and
     print what gets injected.
2. Pull real failure cases from `val.json:failure_trajectories[]` to test.
3. Try 2–3 variants and compare before picking the best.
4. Delete `/tmp/` scripts when done.

Pure-SYSTEM_PROMPT-only candidates may skip prototyping.

### Step 3: Implement

For each candidate slot:

1. Copy `harnesses/<accepted>/harness.py` to `harnesses/<name>/harness.py`
   as your starting point. Make targeted changes only.
2. Validate syntax (cheap, catches `SyntaxError`):
   `python -c "import ast; ast.parse(open('harnesses/<name>/harness.py').read()); print('OK')"`
3. Visually re-read the file to confirm `class CandidateEnv(SearchR1Env)`
   exists and the `from environments.search_r1.base_harness import ...`
   line is present.

### Step 4: Self-critique — MANDATORY

Re-read your written `harness.py`. Ask:

- Does this introduce a genuinely NEW mechanism, or is it a parameter
  variant of base / an existing harness in `evolution_summary.jsonl`?
  - Diff is a single numeric constant changed → parameter variant. REWRITE.
  - `search()` / `env_response()` body byte-identical to parent except
    for constants → parameter variant. REWRITE.
  - SYSTEM_PROMPT must be substantively rewritten (not a single-word swap)
    to count as a real change.
- Does it match the SCHEMA CONTRACT (`query_list: list[str]`, `maxItems=1`)?
- Does my `search()` return `json.dumps({"result": ...})` (not raw string)?
- Does my retrieval payload include `return_scores=True`, the **per-call
  `max_doc_tokens`** (forwarded from the `search()` arg, default 200), and
  `tokenizer_name="Qwen/Qwen3.5-2B"`? Hardcoding `200` instead of the
  variable defeats axis-D tunability.
- Does it mention any dataset/subset name, hardcoded entity, or
  question-text condition? (If yes → REWRITE; see Anti-overfitting rules.)

### Step 5: Write pending_eval.json

Write to the working-directory root:

```json
{
  "candidates": [
    {
      "name": "<slot_name>",
      "hypothesis": "<one falsifiable claim about what should improve>",
      "changes": "<specific description of what you changed>",
      "axis": "<A|B|C|D|E|F>",
      "components": ["<short-tag-1>", "<short-tag-2>"]
    }
  ]
}
```

`axis` uses the letter from "Exploitation axes". `components` are short
mechanism-tags you choose for cross-iteration analysis (e.g. naming the
specific technique applied, not naming any dataset / subset / question).

Output line at the end: `CANDIDATES: <name_1>, <name_2>, ...`

## CANDIDATE FILE STRUCTURE

```python
# harnesses/{name}/harness.py
from environments.search_r1.base_harness import SearchR1Env, search

SYSTEM_PROMPT = "..."
USER_PROMPT_TEMPLATE = "... Question: {question}"

class CandidateEnv(SearchR1Env):
    SYSTEM_PROMPT = SYSTEM_PROMPT
    USER_PROMPT_TEMPLATE = USER_PROMPT_TEMPLATE
    TOOLS = [search]

    # Optional: override env_response for custom nudge / dedup / reformat
    # async def env_response(self, messages, state, **kwargs): ...
```

To redefine `search`: see `base_harness.py:search` for the reference (it
already encodes the required payload shape, lenient multi-query clamp, and
JSON-wrapped return). Copy-then-edit; do NOT write `search` from scratch.

To override `env_response`: see `base_harness.py:env_response` for the
signature.