Mint Eval

mint-eval

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback — Wang et al. (2023) (arXiv:2309.10691, 2023)

What this evaluates

Evaluates large language models' ability to solve tasks using external tools across multiple interaction turns, and their capacity to leverage natural language feedback to improve performance. It also measures the rate of improvement per turn and identifies failure patterns like formatting issues or training data artifacts.

Datasets

• MINT — total ?; splits: test (-1)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Percentage of task instances solved successfully given an interaction limit k. SR_k is measured at each turn k ∈ [1,5].
• Δ_tools — range: percent
  • Improvement rate per additional interaction turn, estimated as the slope b from least-square regression min_{b,a} Σ(b·k + a - SR_k)^2.
• Δ_feedback — range: percent
  • Performance difference after receiving natural language feedback: SR_5^feedback - SR_5.

Input / output format

Input: Task descriptions requiring tool use, presented sequentially in a multi-turn chat interface. The model receives the initial task, previous tool execution results, and optionally simulated natural language feedback from GPT-4.

Output: Tool calls and responses. The model must generate parsable outputs following specific formatting instructions (e.g., wrapping code in tags), though the paper notes some models fail to adhere to these formats.

Scoring recipe

def compute_sr(predictions, gold, k):
    successful = sum(1 for p, g in zip(predictions[:k], gold) if p == g)
    return (successful / len(gold)) * 100

def compute_delta_tools(sr_values, k_values):
    n = len(k_values)
    sum_k = sum(k_values)
    sum_sr = sum(sr_values)
    sum_k_sr = sum(k * sr for k, sr in zip(k_values, sr_values))
    sum_k2 = sum(k**2 for k in k_values)
    b = (n * sum_k_sr - sum_k * sum_sr) / (n * sum_k2 - sum_k**2)
    return b

def compute_delta_feedback(sr_no_fb, sr_with_fb):
    return sr_with_fb - sr_no_fb

Common pitfalls

• Models often fail to produce parsable output or follow formatting instructions (e.g., using [PYTHON] instead of ), which breaks tool execution and artificially lowers scores.
• Training data artifacts (e.g., escaped underscores in Vicuna) can cause syntax errors or degraded performance independent of the model's actual reasoning capability.
• Assuming single-turn task-solving strength predicts multi-turn success; the paper explicitly finds they do not translate.
• RLHF and SIFT often degrade multi-turn interaction performance, contrary to the expectation that alignment improves capability.

Evidence (verbatim from paper)

Metric. We consider Success Rate $SR$ as our evaluation metric, which measures the percentage of successful task instances. For interaction limit $k$, we start from scratch and allow each LLM to interact up to the $k$-th turn and measure their corresponding $SR_k$. Unless otherwise noted, we limit $k \in [1,5]$ where $k = 1$ means no interaction and $k = 5$ maximizes interaction turns within most modern LLMs' context window (4,096 tokens).

Citation

@misc{wang2023mint,
  title={MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2309.10691}
}

• arXiv: 2309.10691