GAIA-specific benchmark guidance. Public-writeup-sourced techniques (markdown browser, file inspector, code-action agent, planning, multi-agent decomposition, query refinement, answer-format guard, early-commit nudge, prompt caching, majority voting) mapped to HarnessX's four levers (config / control / action / instruction), plus a catalogue of common GAIA failure modes (A-H). Use when forming a hypothesis or scanning for which intervention fits a pattern you see in trajectories.
GAIA-specific benchmark guidance. Public-writeup-sourced techniques (markdown browser, file inspector, code-action agent, planning, multi-agent decomposition, query refinement, answer-format guard, early-commit nudge, prompt caching, majority voting) mapped to HarnessX's four levers (config / control / action / instruction), plus a catalogue of common GAIA failure modes (A-H). Use when forming a hypothesis or scanning for which intervention fits a pattern you see in trajectories.
GAIA: Benchmark-specific Guidance
Task Structure
Per the GAIA paper's own frequency table, in decreasing order of
how often each capability is needed:
Web browsing — visit a specific page, not just read a search
snippet
Code execution — compute from retrieved data (pandas, math)
Multi-modal understanding — images, audio, video, PDFs with
figures
Long-horizon planning — Level 3 tasks can require 20-50 steps
Common question shapes: "According to , what is ?",
"How many X in Y between Z1-Z2?", riddles, reversed/encoded text,
embedded-file computations, YouTube video questions.
Authoritative pass/fail from the external pipeline evaluator
eval_score
Authoritative numeric score (0.0 / 1.0 for exact-match)
The dataset's expected answer and the evaluator's textual reason are both intentionally withheld from the frontmatter — the meta-agent sees only pass/fail and the numeric score. For the why behind a failure, read the trajectory body and the optional judge_* fields below.
Opinion-only frontmatter fields (present only when the LLM judge processor ran):
Field
What it tells you
extracted_answer
What the judge's extractor captured
judge_verdict
Opinion only: plausible / unsupported /
hedging / format_wrong / refused /
no_answer / judge_error
judge_cause
Judge's hypothesis for why the run went wrong
judge_missing
Capability the judge thinks was absent
judge_lesson
One-line takeaway suggested by the judge
When eval_passed and judge_verdict disagree, trust eval_* for
pass/fail accounting and use judge_* to probe why.
judge_error is the judge itself failing (timeout, bad JSON), not
the agent — do not build a candidate off those rows.
(Generic trajectory-reading methodology — lens × lever × intent,
three-variant retroactive check, systemic-vs-idiosyncratic filter,
candidate schema — all live in the analyze skill. This playbook
only layers GAIA-specific patterns on top. In particular the
"would it pass with unlimited steps + perfect discipline?"
question is just Variant-A retroactive check with GAIA framing.)
When diagnosis points at a missing capability
GAIA is hard on purpose: many failing tasks would still fail with
more steps, a better prompt, or a tighter loop guard, because the
agent literally does not have the capability the task requires.
These are your highest-leverage opportunities — answer the second
question above honestly ("with unlimited steps and perfect
discipline, would the final state still lack the answer?") and when
the answer is yes, the default response is to author a new tool,
not to nudge the model.
Signals that you are in a capability gap (not a behaviour gap):
judge_missing_capability.present = true with a non-empty
summary — the judge is naming the shape of the missing tool for
you. Use summary as a starting sketch, not as a spec.
judge_missing is non-empty and repeats across a cluster of
failing tasks (e.g. "PDF table parser" on 3+ rows, "JS-rendered
page browser" on 4+ rows). Repetition is evidence the gap is a
class, not a one-off.
Failure modes E (raw bytes), F (missing modality), G (Wayback /
non-HTML endpoints), and most H (oversized context) post-mortems
resolve to new tools rather than processors. When you see these,
go straight to Action-layer candidates.
The trajectory shows many retries against the same endpoint, or
the agent paraphrases instead of quoting — both are "tool returned
nothing usable", not "agent chose badly".
How to act on it:
Read _meta_scratch/CONTEXT.md when present — it's the machine-
rendered lever scoreboard from the journal, so you can see at a
glance whether instruction/control has already been tried on each
task without lift. Use its hypothesis history as the seed for
steps 2-4 below.
Read reference skill in the meta-harness workspace — it has the
authoring mechanics (shape spec in TOOL_SPEC.md before code,
stdlib-first impl, config wiring, end-of-turn checklist). The
replay gate that runs after end_turn is the integration oracle —
no self-written probe.
Consult the "GAIA capability classes" table later in this playbook
— match your cluster's judge_missing_capability summaries to a
row, use its "Shape hint" as a framing for your TOOL_SPEC.md.
Do not copy the hint verbatim; your inputs differ from past teams'.
Follow the recipe's checklist. Re-run the cluster on the new tool
and check the gap closed. If judge_missing_capability.present
still fires on those rows, the tool shape was wrong — revise
before declaring the candidate.
Conversely, do not reach for a new tool when:
judge_missing_capability.present = false — judge already said
the problem is behavioural.
The answer is present in the trajectory body but the FINAL ANSWER: marker / format is wrong (mode C). That is a Control
layer fix.
A single existing tool already covers the capability and only the
instruction to use it is missing. That is an Instruction lever.
Authoring a tool to dodge a prompt fix inflates the action surface
without lifting the score. Authoring a prompt fix to dodge an
obvious capability gap keeps the benchmark locked at the ceiling of
what the current tools can reach.
Known Techniques That Improve GAIA Scores
These are generic GAIA wisdom from public writeups by teams scoring
40-75% on the leaderboard. Each is mapped to the lever you'd use in
HarnessX.
Technique
Layer in HarnessX
Evidence source
Approx. lift
Markdown-view web browser with visit_page / page_down / find_in_page / scroll_to (agent sees the rendered page as paginated structured markdown) instead of raw-HTML WebFetch that pollutes context
Action — new tool
HF "beating GAIA" (Autogen browser), H2O
big — was a main lever taking Transformers Agents from mid-pack to #1; see capability class Stateful browser automation
Specialized file inspector with per-extension handlers: .pdf → text extract, .xlsx/.csv → pandas, .mp3/.wav → ASR, .jpg/.png → captioner/OCR
Action — new tool
HF "beating GAIA", H2O, hetline
large — many questions depend on attached files that default read_file returns as unparsed bytes; see capability classes File-format parsers / Structured spreadsheet reader / Audio transcription / Vision QA
Code-Agent style — agent writes Python snippets (not JSON tool calls) executed in a sandboxed interpreter; loops, math, and dataframe ops compose in one step
Python-action agents beat JSON-action on the same model
Planning / fact-list regeneration every N steps — summarize known facts, rewrite plan; crucially drop the previous plan from context so the model re-evaluates rather than rubber-stamping
Control — new before_model processor
HF "beating GAIA" (N=2 manager, N=5 sub-agent)
moderate; author note: including the prior plan in context lowered the score
Manager + research sub-agent — manager agent delegates research to a sub-agent that returns only clean summaries; keeps manager context clean
Control (via existing spawn_subagent) or Action
HF "beating GAIA", JoyAgent 4-role split
large on multi-hop questions where context cleanliness matters
Multiple search providers wired in (Google CSE + Bing + DuckDuckGo + Tavily) so one outage doesn't brick the agent
Action — new tool(s)
LinkedIn empirical study, H2O
moderate — WebSearch outages are a common failure signal
LLM query refinement — rewrite the user question into 2-3 optimized search queries before hitting the retrieval backend
Instruction + Control
LinkedIn empirical study
modest; compounds with multi-provider
Forced FINAL ANSWER: format guard — on_task_end processor that refuses to stop until marker is present; if missing, triggers one more model turn
Control — new processor
H2O (output verification), GAIA grading contract
+2-4pp on "body has answer, no marker" failures
Early-commit nudge near step budget — prompt section or before_model processor injecting "commit now" when remaining_steps ≤ 3
Instruction or Control
well-known fix for budget_exceeded → no_answer
moderate on Level 2-3 tasks with long chains
Prompt caching on the provider side — GAIA agents re-read the same system prompt every step; caching cuts cost 2-10× without behavior change
Configuration — provider kwargs
H2O (Sonnet 3.5 with prompt caching)
cost-only; no accuracy lift
Majority voting over N samples (typically 3) — run each task multiple times, take the mode
Control — orchestration
H2O (N=3 at max accuracy)
+3-5pp at 3× cost
Right-size the model per task — reasoning models (o1/o3, reasoning_effort=high) are often overkill for GAIA's light-reasoning questions; non-reasoning variants or effort=low/minimal can beat them on latency, cost, and loops
Configuration
hetline 2025-10 (GPT-4o beat GPT-5 on GAIA via fewer graph-depth loops)
context-dependent; worth testing
Don't just map failures to existing tools. The table is a menu
of techniques with known lifts — but GAIA's ceiling on any one
scaffold is set by the tools the scaffold has. Teams that broke
through 60%+ did it by writing new tools that did not exist before
(HuggingFace's markdown browser, their per-extension file inspector).
When a cluster of failures points at a missing capability your
current Action list does not cover, the right move is to author a
new tool sized to that cluster — see
### When diagnosis points at a missing capability above. Pattern-
matching a hard task to the closest-fitting existing tool is how
benchmarks stall.
GAIA capability classes that have unblocked past systems
When the journal flags a cluster and you write _meta_scratch/TOOL_SPEC.md
(per the reference skill's "Before you type — write the shape spec"
section), it helps to know which tool classes have historically
unblocked top teams on GAIA. This is
vocabulary, not a catalogue to copy from.
The pattern is: match your cluster's judge_missing_capability
summaries to one of the classes below, then write a SHAPE from YOUR
failing trajectory inputs. Do not copy implementations from elsewhere
— those were shaped for different input distributions than this run's.
snapshot(url: str, date: str) -> str — Wayback CDX lookup → body of closest capture
mode G when questions reference "the page as of date X" or historical content
h2o: ~2pp on history/archive tasks
Filesystem + shell helper
(already covered by builtin Bash — retune / loosen allowlist before authoring)
one-off text extraction, ad-hoc curl to REST endpoints, file format conversion
none — retune existing Bash
File-format parsers (per extension)
pdf_text(url) -> str / docx_text / mp3_text / … — one tool per format, scoped
mode E when answer is inside a specific binary/semi-structured format
HF per-extension file inspector: the single largest observed lift on attachment-heavy tasks
How to use this table when writing TOOL_SPEC.md
Read your cluster's judge_missing_capability summaries from the
trajectory frontmatter.
Skim this table — which row's "Shape hint" most closely matches what
the summaries describe?
Use the matched row's shape as a starting frame for your
TOOL_SPEC.md. Fill the spec from your cluster's real input
examples (per the reference skill §"Before you type — write
the shape spec"), not from this table's hints.
If no row matches cleanly — author anyway. This table is not
exhaustive. GAIA surfaces novel capability gaps regularly; that's
what evolution is for.
What this table is NOT
Not a list of tools to install (do not author all 10)
Not a checklist ("GAIA needs all classes, let's ship them")
Not an implementation reference (the "Shape hint" is a sketch, not
a spec — your real inputs complete it)
Not a promise the listed lifts will apply to your specific run (past
teams had different input distributions)
Common GAIA Failure Modes (from public post-mortems)
Patterns that appear across many different scaffolds and teams —
not tied to any one round's evidence.
A. Skips the tool, answers from memory
Agent reads the question, believes it knows the answer from training
data, emits it without calling any tool. Often wrong on recent or
long-tail facts.
Signal: very low step count + wrong answer
Fix direction: before_model processor requiring ≥1
information-gathering tool call before an answer is accepted;
or instruction-layer rule "never answer a factual question without
a tool call"
Not this mode when: the agent's single tool call returned
nothing useful and the answer came from training-data recall as
a fallback — that is a capability or retrieval gap, not an
over-confidence pattern. A forced tool call won't help if the
tool it forces has no data to return.
B. Search-snippet answer instead of primary source
Google/Bing snippet contains a non-canonical form (diminutive,
truncated, translated, abbreviated); agent commits it verbatim.
Signal: passes on easy questions but misses exact-match on
named entities
Fix direction: instruction rule "proper-noun answers must come
from a WebFetch of the primary page, not from a snippet"; or a
control-layer guard that blocks commit when the extracted answer
is a proper noun and no WebFetch to the canonical source was made
C. Deliberation without commitment
Agent finds a plausible answer mid-run, keeps searching for
confirmation, hits max_steps with no FINAL ANSWER: emitted.
Signal: exit_reason=budget_exceeded, extracted_answer ends
mid-sentence or is a question
Fix direction: late-step commit nudge (see table above)
Not this mode when: the trajectory's final output has no
answer at all because retrieved data was unparseable or absent
(raw bytes, empty results, HTTP errors, endless redirects). A
commit-nudge cannot help if there is nothing to commit — that
is an upstream capability or retrieval failure. The tell: scan
the last 30-40 lines of the body; if the agent is still saying
"let me try X" / "the page shows..." without any concrete
answer-candidate, do not apply this mode.
D. Recursion / graph-depth loops
Reasoning models especially tend to re-enter the same subproblem,
looping until a depth limit terminates them without output.
Signal: many repeated tool calls with tiny variations;
exit_reason=loop_detected or budget_exceeded with no committed
answer
Fix direction: tighter LoopDetectionProcessor matching on
normalized tool input; or reasoning_effort=minimal / swap to a
non-reasoning model
Not this mode when: the loop is the consequence of an
earlier tool returning no useful output. The agent keeps
retrying because it has no progress signal to move past — a
tighter loop detector will just end the run sooner without the
answer. The lever is one step upstream: whatever the retried
tool was meant to return, it did not.
E. Raw bytes from file / PDF
read_file / WebFetch returns bytes or raw HTML; agent
"describes" the content vaguely ("the document discusses...")
without actually parsing it.
Signal: the task expects a specific value found inside a
binary or semi-structured document (PDF, XLSX, image, audio).
The agent's body shows paraphrase rather than quoted content,
or many retries against the same URL. This does NOT require
the question to name a file attachment — a question that ends
up pointing at a document still qualifies.
Fix direction: after_tool processor that parses PDF/XLSX
bytes transparently; or a specialized file-inspector tool
Not this mode when: the document was read successfully and
the error is in how the agent interpreted clearly-parsed text
(wrong section, arithmetic error, misread table). That is a
reasoning problem, not a parsing one.
F. Missing modality
Question needs video or audio understanding; agent tries
transcript-only; misses visual info.
Signal: YouTube / video tasks fail with empty or
confident-wrong answers
Fix direction: download + frame-extraction tool, or explicit
fallback rule ("if transcript unavailable, report it and stop; do
not guess")
G. Wayback Machine / paywalled content
Playwright-based WebFetch fails on non-HTML endpoints like the
Wayback CDX API or paywalled paper PDFs. Agent retries then guesses.
Signal: tool_error_counts includes WebFetch; body shows
archive.org or paywalled URLs
Fix direction: use curl via Bash for REST/API endpoints
(bypasses Playwright); or dedicated wayback_fetch tool
Not this mode when: WebFetch succeeded and returned
content, but the content is a CAPTCHA, an access-wall page, or
a partial snippet rather than the actual document. That is
mode E (needs a parser / different fetch path), not a tooling
mismatch with Playwright.
H. Oversized context pollution
Agent fetches a 200KB page, stuffs the full text into context,
loses focus.
Signal: total_tokens spikes on 1-2 tasks without
proportional result
Fix direction: markdown-view browser that paginates, or a
sub-agent that summarizes before returning
Not this mode when: the large fetch was the agent's only
path to the data and the content is legitimately relevant. A
markdown browser or sub-agent summarizer is the real fix, not
context compaction — compressing the only useful context makes
the failure silent rather than preventing it.
Red Flags — These Are Pipeline Bugs, Not Agent Bugs
Do not build an agent-layer candidate off these; record in
_meta_scratch/NEEDS_FROM_HUMAN.md instead:
extracted_answer == "" but body contains non-trivial text →
answer-extractor regex bug
judge_verdict: judge_error → judge failed (timeout, bad JSON)
exit_reason: error with no traceback anywhere in the body → run
loop swallowed an exception
