TS Insights Eval

ts-insights-eval

Insight Miner: A Time Series Analysis Dataset for Cross-Domain Alignment with Natural Language — Zhang et al. (2025) (arXiv:2512.11251, 2025)

What this evaluates

Evaluates the ability of large multimodal models to generate accurate, domain-agnostic natural language descriptions of time series trends. It probes cross-modal alignment between visual time series plots (or extracted features) and textual trend explanations.

Datasets

• TS-Insights — total ?; splits: test (69), holdout (50)

Metrics

• final_score (primary) — range: [0, 1]
  • Human experts score each generated description on a 0-2 scale (2: matches original time series, 1: partially correct, 0: not correct). The final score is the sum of all expert scores across all 119 samples, normalized to [0, 1] by dividing by the maximum possible score (119 samples × 3 experts × 2 points).

Input / output format

Input: Time series window visualized as a line plot (for Vision models) or extracted statistical features (for Engineering GPT).

Output: A natural language description of the time series trend.

Scoring recipe

total_score = 0
max_score = 119 * 3 * 2  # samples * experts * max_points
for sample in test_samples + holdout_samples:
    for expert in experts:
        score = expert_score(description, ground_truth_ts)  # 0, 1, or 2
        total_score += score
final_metric = total_score / max_score

Common pitfalls

• Normalization denominator is not explicitly stated in the text but inferred as total_samples * num_experts * max_points.
• Human scoring introduces subjectivity; descriptions are shuffled per sample to mitigate order bias, but inter-rater reliability is not reported.
• The 'holdout' set consists of entirely unseen datasets rather than a standard train/val/test split, making generalization evaluation distinct from in-distribution test evaluation.

Evidence (verbatim from paper)

For each of the 119 samples, we generate one description using each of the above models, and ask three domain experts to manually score the descriptions generated. When presented to the domain expert, the descriptions from different models are shuffled in a random order for each sample. A score of 2 is given if the description matches the original time series, a score of 1 is given if the description is partially correct, and a score of zero is given if the description is not correct. We sum the scores from all human evaluators for all test (holdout) samples and normalize it to $0-1$ to produce the final score for each model.

Citation

@misc{zhang2025insightminer,
  title={Insight Miner: A Time Series Analysis Dataset for Cross-Domain Alignment with Natural Language},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2512.11251}
}

• arXiv: 2512.11251