Fara 7b Agentic Eval

fara-7b-agentic-eval

Fara-7B: An Efficient Agentic Model for Computer Use — Ahmed Awadallah et al. (2025) (arXiv:2511.19663, 2025)

What this evaluates

This evaluation probes the agentic capabilities of computer-use models by measuring their ability to complete multi-step web browsing and task-completion tasks on live websites. It assesses both functional success rates and operational efficiency, including token usage, cost, and interaction length.

Datasets

• WebVoyager — total ?; splits: test (-1)
• Online-Mind2Web — total ?; splits: test (-1)
• DeepShop — total ?; splits: test (-1)
• WebTailBench — total 609; splits: test (609)
• ScreenSpot — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • The percentage of tasks successfully completed out of the total number of tasks. Calculated by averaging results over three independent runs, with trajectories capped at 100 steps.
• cost per task — range: other
  • The average monetary cost in USD required to complete a single task, derived from token pricing and action counts.
• actions per task — range: other
  • The average number of low-level browser actions executed by the model to complete a task.

Input / output format

Input: Screenshot of the current browser/UI state paired with the task instruction/prompt.

Output: Low-level browser actions (e.g., click, type, scroll) or agent step commands.

Scoring recipe

def compute_success_rate(tasks, judge_model):
    successful = 0
    for task in tasks:
        trajectory = run_agent(task.prompt, max_steps=100)
        if trajectory.failed_due_to_env_error:
            trajectory = retry_up_to_5_times(trajectory)
        if judge_model.evaluate(trajectory, task.gold_criteria):
            successful += 1
    return (successful / len(tasks)) * 100
# Average over 3 independent runs for final metric.

Common pitfalls

• Live websites change daily, causing high variance; evaluations must be averaged over multiple runs and only retry on environment errors, not incorrect completions.
• Time-sensitive tasks become stale; requires filtering impossible tasks or modifying future dates to current ones to ensure solvability.
• LLM-as-a-judge success rates often overestimate human-verified accuracy due to prompt and model misalignment.

Evidence (verbatim from paper)

We evaluate our agent on three popular benchmarks against live websites: WebVoyager, Online Mind2Web, and DeepShop, as well as our WebTailBench. For ascertaining success rate, we retain the same prompts, llm-as-a-judge model type, and procedure published with each benchmark. Namely, we use GPT-4o along with the official respective prompts in the LLM-based judge for WebVoyager and Deepshop, and o4-mini as the LLM-based judge for Online-Mind2Web and WebTailBench.

Citation

@misc{awadallah2025fara7b,
  title={Fara-7B: An Efficient Agentic Model for Computer Use},
  author={Ahmed Awadallah et al. (2025)},
  year={2025},
  note={arXiv:2511.19663}
}

• arXiv: 2511.19663