Text2analysis Eval

text2analysis-eval

Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries — He et al. (2023) (arXiv:2312.13671, 2023)

What this evaluates

Evaluates large language models on table question answering with advanced data analysis tasks, including forecasting and chart generation, as well as unclear queries that lack explicit parameters. Probes the model's ability to perform semantic parsing, infer missing parameters, generate executable analysis code, and reason about data visualization.

Datasets

• Text2Analysis — total 2249; splits: test (-1); repo https://github.com/microsoft/Text2Analysis

Metrics

• ECR (primary) — range: [0, 1]
  • Executable Code Ratio: the proportion of generated code snippets that execute without runtime errors.
• pass@1 (primary) — range: [0, 1]
  • Pass rate: the proportion of generated code that executes successfully and produces the correct result matching the gold answer.
• CORR — range: [0, 1]
  • Correlation coefficient between predicted and actual values for forecasting tasks. Absolute value closer to 1 indicates better performance.
• RMSE — range: other
  • Root Mean Square Error between predicted and actual values for forecasting tasks. Lower is better.
• MAE — range: other
  • Mean Absolute Error between predicted and actual values for forecasting tasks. Lower is better.
• MedAE — range: other
  • Median Absolute Error between predicted and actual values for forecasting tasks. Lower is better.

Input / output format

Input: HTML table representation, natural language query, constraints on allowed code generation libraries, and requirements for result formatting.

Output: Python code (or equivalent script) that performs the requested data analysis, computes results, and optionally generates visualizations.

Scoring recipe

def score(predictions, golds):
    ecr_scores = []
    pass_scores = []
    for pred, gold in zip(predictions, golds):
        try:
            exec(pred.code)
            ecr_scores.append(1)
            pass_scores.append(1 if pred.result == gold.result else 0)
        except Exception:
            ecr_scores.append(0)
            pass_scores.append(0)
    # Forecasting regression metrics
    corr = pearsonr(pred.result, gold.result)
    rmse = sqrt(mean_squared_error(gold.result, pred.result))
    mae = mean_absolute_error(gold.result, pred.result)
    medae = median_absolute_error(gold.result, pred.result)
    return {
        'ECR': sum(ecr_scores) / len(ecr_scores),
        'pass@1': sum(pass_scores) / len(pass_scores),
        'CORR': corr, 'RMSE': rmse, 'MAE': mae, 'MedAE': medae
    }

Common pitfalls

• Models frequently fail to parse unclear queries that lack explicit parameters, causing significant drops in ECR and pass rates, especially for chart generation.
• Forecasting tasks require both correct code generation and appropriate statistical model selection/parameter tuning; baselines often generate syntactically correct but semantically flawed code.
• Tabular models excel at simple value lookup but struggle with complex pivot operations and multi-step calculations required by the benchmark.

Evidence (verbatim from paper)

As shown in Table 2, overall experimental results demonstrate that GPT-4 outperforms other models. It achieves the highest ECR on the majority of tasks and the highest pass rate across all tasks.

Citation

@misc{he2023text2analysis,
  title={Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2312.13671}
}

• arXiv: 2312.13671