Financial Agent Eval

financial-agent-eval

Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms — Wang et al. (2026) (arXiv:2604.19299, 2026)

What this evaluates

Evaluates the performance, efficiency, and robustness of small language models (≤10B parameters) under three agent paradigms: base prompting, single-agent tool use, and multi-agent collaboration. It measures how architectural complexity impacts accuracy, latency, and reliability across diverse financial tasks.

Datasets

• Financial Agent Benchmark (20 datasets) — total 1000; splits: test (1000)

Metrics

• Completion Rate — range: [0, 1]
  • Proportion of samples returning a valid response without runtime errors, timeouts, or malformed outputs. Formula: # successful responses / # total samples.
• Average Latency — range: other
  • Mean end-to-end inference time per input sample, including all intermediate reasoning and agent interactions.
• Normalized Response Quality (NRQ) — range: [0, 1]
  • Relative improvement over the Base SLM for each dataset, adjusted so that higher-is-better and lower-is-better metrics are aligned in direction.
• Composite Effectiveness Score (primary) — range: other
  • Standardized Z-score aggregating performance across heterogeneous datasets. Formula: Z_c = (1/N) * sum((X_i - mu_i) / sigma_i), where mu and sigma are computed across datasets for the same model.
• Leading Advantage — range: percent
  • Relative gap between the best and second-best performing architectures. Formula: alpha = (s_best - s_second) / (|s_second| + epsilon) * 100%.

Input / output format

Input: Financial task prompts/questions. For agent paradigms, includes tool definitions (calculator, wiki search, web search) and system instructions for the think-act-observe cycle.

Output: Final textual answer or prediction. For agent systems, includes intermediate reasoning steps and tool calls, but evaluation focuses on the final response.

Scoring recipe

def compute_metrics(predictions, golds, latencies, dataset_metrics):
    # Completion Rate
    valid = [1 for p in predictions if p is not None and not is_malformed(p)]
    completion_rate = sum(valid) / len(predictions)
    
    # Composite Effectiveness Score (Z_c)
    z_scores = []
    for i, (pred, gold) in enumerate(zip(predictions, golds)):
        x_i = dataset_metrics[i](pred, gold)  # Task-specific metric
        z_i = (x_i - mu[i]) / sigma[i]         # mu, sigma from all models on dataset i
        z_scores.append(z_i)
    composite_score = sum(z_scores) / len(z_scores)
    
    return completion_rate, sum(latencies)/len(latencies), composite_score

Common pitfalls

• Task-specific metrics (e.g., accuracy, F1) are used but not explicitly defined in the text; evaluators must infer them from the original dataset papers.
• The Composite Effectiveness Score requires computing dataset-level mean and standard deviation across all evaluated models before normalizing, not per-model statistics.
• Agent interactions are strictly capped at 5 turns, which artificially bounds latency and multi-agent coordination overhead.

Evidence (verbatim from paper)

To evaluate agent paradigms under realistic deployment constraints, we adopt a set of metrics that jointly capture effectiveness, efficiency, and robustness. Standard task-specific metrics are used where appropriate, but they are not redefined here due to space constraints. Below, we describe the metrics defined for this study. Completion Rate measures the robustness of a system in practical deployment settings. We define it as the proportion of samples for which the system returns a valid response without runtime errors, timeouts, or malformed outputs: Completion Rate = # of successful responses / # of total samples. This metric reflects the reliability of a paradigm beyond its nominal task performance.

Citation

@misc{wang2026rethinking,
  title={Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.19299}
}

• arXiv: 2604.19299