Tsmd F1 Eval

tsmd-f1-eval

LoCoMotif: Discovering time-warped motifs in time series — Van Wesenbeeck et al. (2023) (arXiv:2311.17582, 2023)

What this evaluates

Evaluates the quality of discovered motif sets in time series by measuring alignment with ground truth segments. It accounts for variable-length patterns and time warping while penalizing both false discoveries and missed patterns.

Datasets

• TSMD benchmark datasets — total ?; splits: (unstated)

Metrics

• F1-score (primary) — range: [0, 1]
  • Precision and recall are computed via a matching matrix between ground truth and discovered motif sets. Segments are matched if their Jaccard similarity exceeds 0.5. Precision is the ratio of correctly matched segments to all predicted segments; recall is the ratio to all ground truth segments. F1 is the harmonic mean of precision and recall.

Input / output format

Input: Ground truth motif sets G and discovered motif sets M, each comprising non-overlapping time series segments.

Output: Scalar F1-score value between 0 and 1.

Scoring recipe

# Match segments with Jaccard > 0.5
matches = greedy_match(gold_segments, pred_segments, threshold=0.5)
# Build matching matrix M* of size (k'+1) x (k+1)
M_star = build_matching_matrix(gold_sets, pred_sets, matches)
k_min = min(len(gold_sets), len(pred_sets))
tp = sum(M_star[i, i] for i in range(k_min))
pred_total = sum(M_star[i, j] for j in range(k_min) for i in range(len(gold_sets)+1))
gold_total = sum(M_star[i, j] for i in range(len(gold_sets)) for j in range(len(pred_sets)+1))
precision = tp / pred_total if pred_total > 0 else 0
recall = tp / gold_total if gold_total > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1

Common pitfalls

• Prior metrics often only support single motif sets or fixed lengths, failing to capture variable-length patterns.
• Many existing metrics do not penalize false discoveries, leading to inflated quality scores.
• The segment matching threshold is strictly set to a Jaccard similarity of 0.5; using a different threshold alters the matching matrix and final score.

Evidence (verbatim from paper)

To define F1-score, we use the micro-average of precision and recall for multi-class classification (Grandini et al. 2020): $$ \mathrm {p r e c i s i o n} = \frac {\sum_ {i = 1} ^ {\kappa_ {\mathrm {m i n}}} M _ {i , i} ^ {}}{\sum_ {j = 1} ^ {\kappa_ {\mathrm {m i n}}} \left(\sum_ {i = 1} ^ {\kappa^ {\prime} + 1} M _ {i , j} ^ {}\right)} \quad \mathrm {a n d} \quad \mathrm {r e c a l l} = \frac {\sum_ {i = 1} ^ {\kappa_ {\mathrm {m i n}}} M _ {i , i} ^ {}}{\sum_ {i = 1} ^ {\kappa^ {\prime}} \left(\sum_ {j = 1} ^ {\kappa + 1} M _ {i , j} ^ {}\right)} $$ where $\kappa_{\mathrm{min}} = \min (\kappa ,\kappa^{\prime})$ (see Fig. 6b for an example). Our F1-score is defined as the harmonic mean of this precision and recall, and evaluates the quality of $\mathcal{M}$ in all aspects, as precision penalizes unmatched segments in discovered motif sets (that is, false discoveries) and recall penalizes undiscovered segments in GT motif sets.

Citation

@misc{vanwesenbeeck2023locomotif,
  title={LoCoMotif: Discovering time-warped motifs in time series},
  author={Van Wesenbeeck et al. (2023)},
  year={2023},
  note={arXiv:2311.17582}
}

• arXiv: 2311.17582