Time Ra Eval

time-ra-eval

Time-RA: Towards Time Series Reasoning for Anomaly Diagnosis with LLM Feedback — Yang et al. (2025) (arXiv:2507.15066, 2025)

What this evaluates

Evaluates the ability of LLMs and MLLMs to diagnose anomalies in univariate and multivariate time series data. It probes the models' capacity for structured reasoning (generating a 'Thought') and precise action classification ('ActionID') based on raw or visualized temporal data.

Datasets

• RATs40K — total ?; splits: test (-1)

Metrics

• Label Matching F1 (primary) — range: [0, 1]
  • Weighted F1 score computed from precision and recall of exact label matches between predicted and ground truth anomaly labels.
• ActionID Matching F1 — range: [0, 1]
  • Weighted F1 score for exact matching of the predicted anomaly action category against the ground truth.
• Thought Matching (Cosine, TF-IDF, Levenshtein, Token, RCS) — range: [0, 1]
  • Semantic and lexical similarity metrics comparing the generated reasoning text to the ground truth explanation. Includes cosine similarity, TF-IDF similarity, Levenshtein distance, token overlap, and RCS.

Input / output format

Input: Time series data (univariate or multivariate), optionally accompanied by visualized plots, formatted into a fixed instruction template.

Output: A text response containing a structured 'Thought' (reasoning) and 'Action' (anomaly classification/action ID).

Scoring recipe

import re
pred_thought = re.search(r'Thought:\s*(.*)', output, re.DOTALL).group(1).strip()
pred_action = re.search(r'Action:\s*(.*)', output, re.DOTALL).group(1).strip()
pred_label = extract_label(pred_action)
gold_label = extract_label(gold_action)
precision = (pred_label == gold_label).mean()
recall = precision
f1 = 2 * precision * recall / (precision + recall + 1e-8)
score_cosine = cosine_similarity(embed(pred_thought), embed(gold_thought))
score_tfidf = tfidf_similarity(pred_thought, gold_thought)
score_lev = 1 - levenshtein_distance(pred_thought, gold_thought) / max(len(pred_thought), len(gold_thought))
score_token = jaccard(set(pred_thought.split()), set(gold_thought.split()))
score_rcs = compute_rcs(pred_thought, gold_thought)

Common pitfalls

• Regex-based extraction of 'Thought' and 'Action' may fail if the model deviates from the expected formatting, leading to missing or malformed predictions.
• Semantic similarity metrics (Cosine, TF-IDF) for 'Thought Matching' are sensitive to phrasing variations and may not fully capture the quality or correctness of the diagnostic reasoning.
• Performance gains from Supervised Fine-Tuning (SFT) are inconsistent, particularly in complex multivariate scenarios where models may stagnate or regress.

Evidence (verbatim from paper)

To evaluate the model outputs, we design regular expressions to automatically extract the predicted Thought and Action, which will then be compared against the ground truth. Evaluation metrics follow the definition of the Time-RAtask, with the best results in bold and second-best underline. F1 means weighted-F1 score.

Citation

@misc{yang2025timera,
  title={Time-RA: Towards Time Series Reasoning for Anomaly Diagnosis with LLM Feedback},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.15066}
}

• arXiv: 2507.15066