Uinnexus Eval

uinnexus-eval

Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System — Guo et al. (2025) (arXiv:2506.08972, 2025)

What this evaluates

Evaluates mobile agents' ability to complete long-horizon, dependency-rich tasks on real mobile applications. It specifically probes atomic-to-compositional generalization, testing how well agents handle task concatenation, context transitions, and deep analysis across different app types and languages.

Datasets

• UI-NEXUS — total 50; splits: test (50)

Metrics

• Success Rate (primary) — range: percent
  • Percentage of tasks completed successfully out of the total number of tasks. Calculated as (number of successful completions / total tasks) * 100.
• PGR — range: percent
  • Performance Gap Recovered. Measures how much of the gap between weak (direct compositional execution) and strong ceiling (oracle decomposition) performance is closed. Formula: (Atomic-to-compositional performance - Weak performance) / (Strong ceiling performance - Weak performance).
• Inference Latency — range: other
  • Average time taken per agent step to process UI state and output an action.
• Inference Cost — range: other
  • Average monetary cost in USD incurred per agent step.

Input / output format

Input: UI state (screenshots and/or accessibility trees) and natural language task instructions (either atomic subtasks or compositional long-horizon instructions).

Output: Sequential mobile UI actions (e.g., click, swipe, type, back) to interact with the application and complete the given task.

Scoring recipe

def compute_success_rate(predictions, gold):
    successful = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (successful / len(gold)) * 100

def compute_pgr(weak_perf, strong_perf, oracle_perf):
    if oracle_perf == weak_perf:
        return 0.0
    return ((strong_perf - weak_perf) / (oracle_perf - weak_perf)) * 100

Common pitfalls

• Agents frequently fail due to 'Budget Exceeded' or 'Premature' termination rather than actual task failure, especially for agent-as-a-model baselines.
• Chinese online service apps often lack proper accessibility tree support, which artificially penalizes agents that rely on structured UI parsing (e.g., M3A).
• Success Rate is measured end-to-end; partial completion or getting stuck in loops counts as failure, making the metric strict for long-horizon tasks.

Evidence (verbatim from paper)

We evaluate representative mobile agents implemented as agentic workflow and agent-as-a-model. ... UI-NEXUS poses substantial challenges on all five mobile agent baselines, with no agent exceeding task success rate of more than 50% on all subsets. ... Inspired by Weak-to-Strong Generalization [64], we define the Performance Gap Recovered (PGR) as: PGR = (Atomic-to-compositional performance - Weak performance) / (Strong ceiling performance - Weak performance)

Citation

@misc{guo2025uinnexus,
  title={Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2506.08972}
}

• arXiv: 2506.08972