Taskbench Dataset Quality Eval

taskbench-dataset-quality-eval

TaskBench: Benchmarking Large Language Models for Task Automation — Yongliang Shen et al. (2023) (arXiv:2311.18760, 2023)

What this evaluates

Evaluates the quality of synthetically generated task automation instructions and tool invocation graphs. It probes whether the generated data is natural, appropriately complex, and correctly aligned with the underlying tool dependencies.

Datasets

• Hugging Face Tools — total 12217; splits: test (12217)
• Multimedia Tools — total 8904; splits: test (8904)
• Daily Life APIs — total 7150; splits: test (7150)

Metrics

• Naturalness — range: other
  • Human-rated score (1-5) measuring the reasonableness of instructions, including the commonality of dependencies between tools and their alignment with real-world needs.
• Complexity — range: other
  • Human-rated score (1-5) assessing the complexity of instructions, considering factors such as task depth, the number of involved tools, and the relationships between these tools.
• Alignment (primary) — range: other
  • Human-rated score (1-5) measuring how well the tool invocation graphs align with the instructions, i.e., whether the tool invocation graphs can effectively address the user's commands.

Input / output format

Input: A generated user instruction and its corresponding tool invocation graph (including task steps and tool dependencies).

Output: A numerical rating from 1 to 5 for each metric, provided by human domain experts.

Scoring recipe

def score_dataset_quality(samples, human_ratings):
    scores = {'Naturalness': [], 'Complexity': [], 'Alignment': []}
    for sample in samples:
        r = human_ratings[sample.id]
        scores['Naturalness'].append(r['Naturalness'])
        scores['Complexity'].append(r['Complexity'])
        scores['Alignment'].append(r['Alignment'])
    return {k: sum(v)/len(v) for k, v in scores.items()}

Common pitfalls

• Evaluation is performed on a very small subset (50 samples) rather than the full dataset.
• Metrics are entirely subjective and rely on human judgment without automated execution or ground-truth verification.
• Human experts are only calibrated with canonical samples, which may not cover the full distribution of generated instructions.

Evidence (verbatim from paper)

To assess the quality of datasets constructed by Back-Instruct, we designed three metrics in our evaluation criteria. Two measure the quality of instructions, and one evaluates tool invocation graphs: Metrics for Instruction: Naturalness: This metric measures the reasonableness of the instructions, including the commonality of dependencies between tools and their alignment with real-world needs. Complexity: This metric assesses the complexity of the instructions, considering factors such as task depth, the number of involved tools, and the relationships between these tools. Metric for Tool Invocation Graphs: Alignment: Building upon the Feasibility metric, this measures how well the tool invocation graphs align with the instructions, i.e., whether the tool invocation graphs can effectively address the user's commands. Each metric is scored from 1 to 5...

Citation

@misc{shen2023taskbench,
  title={TaskBench: Benchmarking Large Language Models for Task Automation},
  author={Yongliang Shen et al. (2023)},
  year={2023},
  note={arXiv:2311.18760}
}

• arXiv: 2311.18760