Gaia Playbook

GAIA: Benchmark-specific Guidance

Task Structure

Per the GAIA paper's own frequency table, in decreasing order of how often each capability is needed:

1. Web browsing — visit a specific page, not just read a search snippet
2. Code execution — compute from retrieved data (pandas, math)
3. File handling — parse .pdf, .xlsx, .csv, .docx, .mp3, .mp4, .jpg, .zip
4. Multi-modal understanding — images, audio, video, PDFs with figures
5. Long-horizon planning — Level 3 tasks can require 20-50 steps

Common question shapes: "According to , what is ?", "How many X in Y between Z1-Z2?", riddles, reversed/encoded text, embedded-file computations, YouTube video questions.

Analyzing Failures

Primary-evidence frontmatter fields (always reliable):

The dataset's expected answer and the evaluator's textual reason are both intentionally withheld from the frontmatter — the meta-agent sees only pass/fail and the numeric score. For the why behind a failure, read the trajectory body and the optional judge_* fields below.

Opinion-only frontmatter fields (present only when the LLM judge processor ran):

When eval_passed and judge_verdict disagree, trust eval_* for pass/fail accounting and use judge_* to probe why. judge_error is the judge itself failing (timeout, bad JSON), not the agent — do not build a candidate off those rows.

(Generic trajectory-reading methodology — lens × lever × intent, three-variant retroactive check, systemic-vs-idiosyncratic filter, candidate schema — all live in the analyze skill. This playbook only layers GAIA-specific patterns on top. In particular the "would it pass with unlimited steps + perfect discipline?" question is just Variant-A retroactive check with GAIA framing.)

When diagnosis points at a missing capability

GAIA is hard on purpose: many failing tasks would still fail with more steps, a better prompt, or a tighter loop guard, because the agent literally does not have the capability the task requires. These are your highest-leverage opportunities — answer the second question above honestly ("with unlimited steps and perfect discipline, would the final state still lack the answer?") and when the answer is yes, the default response is to author a new tool, not to nudge the model.

Signals that you are in a capability gap (not a behaviour gap):

• judge_missing_capability.present = true with a non-empty summary — the judge is naming the shape of the missing tool for you. Use summary as a starting sketch, not as a spec.
• judge_missing is non-empty and repeats across a cluster of failing tasks (e.g. "PDF table parser" on 3+ rows, "JS-rendered page browser" on 4+ rows). Repetition is evidence the gap is a class, not a one-off.
• Failure modes E (raw bytes), F (missing modality), G (Wayback / non-HTML endpoints), and most H (oversized context) post-mortems resolve to new tools rather than processors. When you see these, go straight to Action-layer candidates.
• The trajectory shows many retries against the same endpoint, or the agent paraphrases instead of quoting — both are "tool returned nothing usable", not "agent chose badly".

How to act on it:

1. Read _meta_scratch/CONTEXT.md when present — it's the machine- rendered lever scoreboard from the journal, so you can see at a glance whether instruction/control has already been tried on each task without lift. Use its hypothesis history as the seed for steps 2-4 below.
2. Read reference skill in the meta-harness workspace — it has the authoring mechanics (shape spec in TOOL_SPEC.md before code, stdlib-first impl, config wiring, end-of-turn checklist). The replay gate that runs after end_turn is the integration oracle — no self-written probe.
3. Consult the "GAIA capability classes" table later in this playbook — match your cluster's judge_missing_capability summaries to a row, use its "Shape hint" as a framing for your TOOL_SPEC.md. Do not copy the hint verbatim; your inputs differ from past teams'.
4. Follow the recipe's checklist. Re-run the cluster on the new tool and check the gap closed. If judge_missing_capability.present still fires on those rows, the tool shape was wrong — revise before declaring the candidate.

Conversely, do not reach for a new tool when:

• judge_missing_capability.present = false — judge already said the problem is behavioural.
• The answer is present in the trajectory body but the FINAL ANSWER: marker / format is wrong (mode C). That is a Control layer fix.
• A single existing tool already covers the capability and only the instruction to use it is missing. That is an Instruction lever.

Authoring a tool to dodge a prompt fix inflates the action surface without lifting the score. Authoring a prompt fix to dodge an obvious capability gap keeps the benchmark locked at the ceiling of what the current tools can reach.

Known Techniques That Improve GAIA Scores

These are generic GAIA wisdom from public writeups by teams scoring 40-75% on the leaderboard. Each is mapped to the lever you'd use in HarnessX.

Don't just map failures to existing tools. The table is a menu of techniques with known lifts — but GAIA's ceiling on any one scaffold is set by the tools the scaffold has. Teams that broke through 60%+ did it by writing new tools that did not exist before (HuggingFace's markdown browser, their per-extension file inspector). When a cluster of failures points at a missing capability your current Action list does not cover, the right move is to author a new tool sized to that cluster — see ### When diagnosis points at a missing capability above. Pattern- matching a hard task to the closest-fitting existing tool is how benchmarks stall.

GAIA capability classes that have unblocked past systems

When the journal flags a cluster and you write _meta_scratch/TOOL_SPEC.md (per the reference skill's "Before you type — write the shape spec" section), it helps to know which tool classes have historically unblocked top teams on GAIA. This is vocabulary, not a catalogue to copy from.

The pattern is: match your cluster's judge_missing_capability summaries to one of the classes below, then write a SHAPE from YOUR failing trajectory inputs. Do not copy implementations from elsewhere — those were shaped for different input distributions than this run's.

How to use this table when writing TOOL_SPEC.md

1. Read your cluster's judge_missing_capability summaries from the trajectory frontmatter.
2. Skim this table — which row's "Shape hint" most closely matches what the summaries describe?
3. Use the matched row's shape as a starting frame for your TOOL_SPEC.md. Fill the spec from your cluster's real input examples (per the reference skill §"Before you type — write the shape spec"), not from this table's hints.
4. If no row matches cleanly — author anyway. This table is not exhaustive. GAIA surfaces novel capability gaps regularly; that's what evolution is for.

What this table is NOT

• Not a list of tools to install (do not author all 10)
• Not a checklist ("GAIA needs all classes, let's ship them")
• Not an implementation reference (the "Shape hint" is a sketch, not a spec — your real inputs complete it)
• Not a promise the listed lifts will apply to your specific run (past teams had different input distributions)

Common GAIA Failure Modes (from public post-mortems)

Patterns that appear across many different scaffolds and teams — not tied to any one round's evidence.

A. Skips the tool, answers from memory

Agent reads the question, believes it knows the answer from training data, emits it without calling any tool. Often wrong on recent or long-tail facts.

• Signal: very low step count + wrong answer
• Fix direction: before_model processor requiring ≥1 information-gathering tool call before an answer is accepted; or instruction-layer rule "never answer a factual question without a tool call"
• Not this mode when: the agent's single tool call returned nothing useful and the answer came from training-data recall as a fallback — that is a capability or retrieval gap, not an over-confidence pattern. A forced tool call won't help if the tool it forces has no data to return.

B. Search-snippet answer instead of primary source

Google/Bing snippet contains a non-canonical form (diminutive, truncated, translated, abbreviated); agent commits it verbatim.

• Signal: passes on easy questions but misses exact-match on named entities
• Fix direction: instruction rule "proper-noun answers must come from a WebFetch of the primary page, not from a snippet"; or a control-layer guard that blocks commit when the extracted answer is a proper noun and no WebFetch to the canonical source was made

C. Deliberation without commitment

Agent finds a plausible answer mid-run, keeps searching for confirmation, hits max_steps with no FINAL ANSWER: emitted.

• Signal: exit_reason=budget_exceeded, extracted_answer ends mid-sentence or is a question
• Fix direction: late-step commit nudge (see table above)
• Not this mode when: the trajectory's final output has no answer at all because retrieved data was unparseable or absent (raw bytes, empty results, HTTP errors, endless redirects). A commit-nudge cannot help if there is nothing to commit — that is an upstream capability or retrieval failure. The tell: scan the last 30-40 lines of the body; if the agent is still saying "let me try X" / "the page shows..." without any concrete answer-candidate, do not apply this mode.

D. Recursion / graph-depth loops

Reasoning models especially tend to re-enter the same subproblem, looping until a depth limit terminates them without output.

• Signal: many repeated tool calls with tiny variations; exit_reason=loop_detected or budget_exceeded with no committed answer
• Fix direction: tighter LoopDetectionProcessor matching on normalized tool input; or reasoning_effort=minimal / swap to a non-reasoning model
• Not this mode when: the loop is the consequence of an earlier tool returning no useful output. The agent keeps retrying because it has no progress signal to move past — a tighter loop detector will just end the run sooner without the answer. The lever is one step upstream: whatever the retried tool was meant to return, it did not.

E. Raw bytes from file / PDF

read_file / WebFetch returns bytes or raw HTML; agent "describes" the content vaguely ("the document discusses...") without actually parsing it.

• Signal: the task expects a specific value found inside a binary or semi-structured document (PDF, XLSX, image, audio). The agent's body shows paraphrase rather than quoted content, or many retries against the same URL. This does NOT require the question to name a file attachment — a question that ends up pointing at a document still qualifies.
• Fix direction: after_tool processor that parses PDF/XLSX bytes transparently; or a specialized file-inspector tool
• Not this mode when: the document was read successfully and the error is in how the agent interpreted clearly-parsed text (wrong section, arithmetic error, misread table). That is a reasoning problem, not a parsing one.

F. Missing modality

Question needs video or audio understanding; agent tries transcript-only; misses visual info.

• Signal: YouTube / video tasks fail with empty or confident-wrong answers
• Fix direction: download + frame-extraction tool, or explicit fallback rule ("if transcript unavailable, report it and stop; do not guess")

G. Wayback Machine / paywalled content

Playwright-based WebFetch fails on non-HTML endpoints like the Wayback CDX API or paywalled paper PDFs. Agent retries then guesses.

• Signal: tool_error_counts includes WebFetch; body shows archive.org or paywalled URLs
• Fix direction: use curl via Bash for REST/API endpoints (bypasses Playwright); or dedicated wayback_fetch tool
• Not this mode when: WebFetch succeeded and returned content, but the content is a CAPTCHA, an access-wall page, or a partial snippet rather than the actual document. That is mode E (needs a parser / different fetch path), not a tooling mismatch with Playwright.

H. Oversized context pollution

Agent fetches a 200KB page, stuffs the full text into context, loses focus.

• Signal: total_tokens spikes on 1-2 tasks without proportional result
• Fix direction: markdown-view browser that paginates, or a sub-agent that summarizes before returning
• Not this mode when: the large fetch was the agent's only path to the data and the content is legitimately relevant. A markdown browser or sub-agent summarizer is the real fix, not context compaction — compressing the only useful context makes the failure silent rather than preventing it.

Red Flags — These Are Pipeline Bugs, Not Agent Bugs

Do not build an agent-layer candidate off these; record in _meta_scratch/NEEDS_FROM_HUMAN.md instead:

• extracted_answer == "" but body contains non-trivial text → answer-extractor regex bug
• judge_verdict: judge_error → judge failed (timeout, bad JSON)
• exit_reason: error with no traceback anywhere in the body → run loop swallowed an exception