Hippocamp Eval

hippocamp-eval

HippoCamp: Benchmarking Contextual Agents on Personal Computers — Yang et al. (2026) (arXiv:2604.01221, 2026)

What this evaluates

Evaluates multimodal agents' ability to reason over personalized, device-scale file systems. It probes long-horizon cross-file retrieval, multimodal perception, and evidence-grounded factual retention under strict profile-isolation constraints.

Datasets

• HippoCamp — total 581; splits: test (581)

Metrics

• accuracy (primary) — range: percent
  • Fraction of generated responses judged correct by an LLM-as-a-judge based on factual alignment, reasoning soundness, and contextual personalization.
• F1 — range: percent
  • Harmonic mean of precision and recall for retrieved evidence files, measuring the balance between retrieving relevant files and avoiding spurious ones.

Input / output format

Input: Natural-language query paired with profile-local access to a multimodal file corpus (up to 42.4 GB across 2K files). Agents may freely explore, search, and perceive the environment.

Output: Free-form generated response (answer/reasoning) and optionally retrieved evidence files or tool-use traces.

Scoring recipe

# For QA tasks
judge_score = llm_judge(query, gold_answer, model_response) # returns 0 or 1
accuracy = sum(judge_score) / total_tasks

# For retrieval tasks
precision = len(retrieved_files & gold_files) / len(retrieved_files)
recall = len(retrieved_files & gold_files) / len(gold_files)
f1 = 2 * (precision * recall) / (precision + recall)

Common pitfalls

• Execution environment heavily impacts results: Dockerized terminal agents vs. native product-grade modes show significant variance and instability, making direct comparison difficult.
• LLM-as-a-judge evaluation replaces strict string matching, which may introduce subjective bias or inconsistency in correctness judgments.
• Strict profile-isolation constraint forbids external retrieval or side-channel metadata, unlike many standard benchmarks that assume open web access.

Evidence (verbatim from paper)

We report overall accuracy, measured as the fraction of responses judged correct. Evidence retrieval evaluation. For tasks requiring document or file retrieval, we assess retrieval quality using recall hit rate and F1 score based on the ground-truth evidence file set. F1 captures the balance between retrieving relevant files and avoiding spurious ones, while recall measures the agent's ability to identify all necessary evidence supporting correct reasoning.

Citation

@misc{yang2026hippocamp,
  title={HippoCamp: Benchmarking Contextual Agents on Personal Computers},
  author={Yang et al. (2026)},
  year={2026},
  note={arXiv:2604.01221}
}

• arXiv: 2604.01221