Filegrambench Eval

filegrambench-eval

FileGram: Grounding Agent Personalization in File-System Behavioral Traces — Shuai Liu et al. (2026) (arXiv:2604.04901, 2026)

What this evaluates

Evaluates AI agents' ability to personalize based on file-system behavioral traces across procedural, semantic, and episodic memory channels. It probes attribute recognition, behavioral inference, anomaly detection, and grounding using simulated, multimodal, and real-world settings.

Datasets

• FileGramBench — total 640; splits: test (-1); repo https://github.com/Synvo-ai/FileGram

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions scaled to 0–100. Multiple-choice tasks use exact match. Open-ended tasks are scored by an LLM judge on a Likert 1–5 scale, then rescaled to 0–100.

Input / output format

Input: Behavioral event logs (text), rendered PDFs/images, or human screen recordings, paired with task-specific queries (multiple-choice or free-form prompts).

Output: Discrete choices for multiple-choice tasks, or free-form text responses for open-ended tasks.

Scoring recipe

def compute_accuracy(predictions, gold, is_open_ended=False):
    if is_open_ended:
        raw = llm_judge_score(predictions, gold) # 1-5
        return ((raw - 1) / 4) * 100
    return 1.0 if predictions == gold else 0.0
# Aggregate: sum(scores) / len(gold) * 100

Common pitfalls

• Assuming multimodal inputs (PDFs/images) automatically improve behavioral discrimination; vision-based methods fail on operation-level statistics like file counts and directory depth.
• Treating simulated clean traces as equivalent to real-world video recordings; the sim-to-real gap causes accuracy to drop to single digits on human screen recordings due to noise and unstructured visual input.
• Relying on narrative summarization during ingestion, which prematurely erases fine-grained behavioral discriminators (e.g., action counts, edit granularity) and flattens distinct user profiles.

Evidence (verbatim from paper)

Table 3: Main results on FileGramBench.All scores are accuracy (%) scaled 0–100. †Open-ended sub-tasks are scored by LLM judge (Likert 1–5, rescaled).

Citation

@misc{liu2026filegram,
  title={FileGram: Grounding Agent Personalization in File-System Behavioral Traces},
  author={Shuai Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.04901}
}

• arXiv: 2604.04901