Creative Fatigue Detection Eval

creative-fatigue-detection-eval

A Path Signature Framework for Detecting Creative Fatigue in Digital Advertising — Shaw (2025) (arXiv:2509.09758, 2025)

What this evaluates

Evaluates change point detection algorithms for identifying the onset of creative fatigue in digital advertising campaigns. It probes early warning capabilities, precision-recall trade-offs, and detection latency against ground-truth performance degradation events.

Datasets

• Synthetic Gradual Decline — total ?; splits: test (-1)
• Synthetic Sharp Decline — total ?; splits: test (-1)

Metrics

• Precision — range: [0, 1]
  • Ratio of correctly detected fatigue change points to all detected change points.
• Recall — range: [0, 1]
  • Ratio of correctly detected fatigue change points to all actual fatigue change points.
• F1-Score — range: [0, 1]
  • Harmonic mean of Precision and Recall.
• Delay (days) (primary) — range: other
  • Detected change date minus ground-truth change date. Negative values indicate early warning.

Input / output format

Input: Daily performance time-series data for digital advertising creatives, transformed into geometric paths via path signature analysis.

Output: Detected change point dates indicating the onset of creative fatigue.

Scoring recipe

def compute_metrics(detected_dates, ground_truth_dates):
    # Match detected to ground truth within a tolerance window
    tp = sum(1 for d in detected_dates if any(abs(d - g) <= tolerance for g in ground_truth_dates))
    precision = tp / len(detected_dates) if detected_dates else 0
    recall = tp / len(ground_truth_dates) if ground_truth_dates else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    delays = [d - g for d, g in zip(detected_dates, ground_truth_dates)]
    avg_delay = sum(delays) / len(delays) if delays else 0
    return precision, recall, f1, avg_delay

Common pitfalls

• Low precision (0.14) is intentional to maximize recall and enable early warnings; misinterpreting it as poor performance ignores the early-warning design goal.
• Negative delay values indicate early detection, not system lag; readers may incorrectly treat them as errors.
• Traditional baselines (MA Crossover, Rolling Regression) fail catastrophically on sharp declines due to violated statistical assumptions, making direct comparison without context misleading.

Evidence (verbatim from paper)

The performance metrics for each method on the two synthetic datasets—one with a gradual decline and one with a sharp decline—are summarised in Table 1. ... For clarity, we define detection delay as the detected change date minus the ground-truth change date; negative values therefore indicate an early warning. The method demonstrates a high recall of 1.00, ensuring it never misses actual fatigue events—critical when each missed detection represents continued wastage. The precision of 0.14 reflects our method’s sensitivity to a broad spectrum of performance changes, not just monotonic decline.

Citation

@misc{shaw2025pathsignature,
  title={A Path Signature Framework for Detecting Creative Fatigue in Digital Advertising},
  author={Shaw (2025)},
  year={2025},
  note={arXiv:2509.09758}
}

• arXiv: 2509.09758