qhjqhj00 prom-eval
Quantitative Evaluation of Motif Sets in Time Series — Van Wesenbeeck et al. (2024) (arXiv:2412.09346, 2024)
What this evaluates
Evaluates how well discovered motif sets in time series approximate ground truth motif sets. It penalizes false positives, false negatives, and redundant motifs without requiring uniform motif lengths or a fixed number of motif sets.
Datasets
- TSMD-Bench — total 2800; splits: test (2800); repo https://github.com/ML-KULeuven/tsdm-evaluation
Metrics
PROM(primary) — range: [0, 1]- Computes precision, recall, and F1-score by optimally matching ground truth and discovered motif sets. It penalizes false positives, false negatives, and redundant motifs based on overlap rates between matched motifs. Exact matching logic and thresholds are detailed in the paper's Section 4.3.
Input / output format
Input: Time series (univariate or multivariate) paired with ground truth motif sets.
Output: Discovered motif sets, where each motif is specified by its start index, length, and subsequence values.
Scoring recipe
def compute_prom_f1(gold, discovered):
matches = find_optimal_matching(gold, discovered)
tp, fp, fn = 0, 0, 0
for g_set, d_set in matches:
for g_motif in g_set:
if any(overlap(g_motif, d) > 0.5 for d in d_set):
tp += 1
else:
fn += 1
for d_motif in d_set:
if not any(overlap(g, d_motif) > 0.5 for g in g_set):
fp += 1
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Prior metrics like Correctness can exceed 1 and reward detecting the same ground truth segment multiple times.
- Score metric is highly sensitive to motif length and number, making cross-dataset comparison difficult.
- PROM does not require uniform motif lengths or fixed cardinalities, unlike many traditional TSMD benchmarks.
Evidence (verbatim from paper)
PROM is a novel, assumption-free evaluation metric for time series motif discovery that quantitatively assesses how well discovered motif sets approximate ground truth motif sets by penalizing false positives, false negatives, and redundant motifs—without requiring uniform motif lengths or fixed numbers of motif sets.
Citation
@misc{vanwesenbeeck2024prom,
title={Quantitative Evaluation of Motif Sets in Time Series},
author={Van Wesenbeeck et al. (2024)},
year={2024},
note={arXiv:2412.09346}
}
- arXiv: 2412.09346