Novelty Detection Eval

novelty-detection-eval

Using temporal IDF for efficient novelty detection in text streams — Karkali et al. (2014) (arXiv:1401.1456, 2014)

What this evaluates

Evaluates a system's ability to identify whether an incoming document contains novel information relative to a recent sliding window of previously seen documents in a text stream, using term specificity rather than pairwise similarity.

Datasets

• Real-world news stream — total ?; splits: test (-1)

Metrics

• precision, recall, F1 (primary) — range: percent
  • Standard novelty detection metrics calculated against human-annotated novelty labels. Precision measures the proportion of correctly identified novel documents among all predicted novel ones. Recall measures the proportion of actual novel documents correctly identified. F1 is the harmonic mean. Computational efficiency (time/memory) is also reported.

Input / output format

Input: A timestamped document represented as a bag-of-words vector, compared against a fixed-size sliding window corpus of previously indexed documents.

Output: A continuous novelty score NS(d^t, C), thresholded to produce a binary decision (novel if NS > θ, else not novel).

Scoring recipe

def novelty_score(doc, corpus):
    N = len(corpus)
    score = 0.0
    for q in unique_terms(doc):
        df_q = count_docs_containing(corpus, q)
        idf = log((N + 1) / (df_q + 0.5))  # smoothed variant
        score += tf(q, doc) * idf
    return score / normalize(doc)

is_novel = novelty_score(new_doc, window) > theta

Common pitfalls

• Sliding window size critically affects performance; too small causes false alarms, too large causes missed updates.
• Corpus for IDF computation is often conflated with the comparison window, though they serve different purposes.
• Handling zero document frequencies requires careful smoothing (e.g., add-half Laplace) to avoid undefined scores in small windows.

Evidence (verbatim from paper)

Experiments on a real-world news stream show superior performance over baseline methods in both precision and efficiency, especially in mobile settings where resource constraints are critical. We define the Novelty Detection (ND) problem as the characterization of an incoming document as novel with respect to a predefined window in the past. In the described context we declare novel a document $d^{t}$ when the corresponding novelty score $NS(d^{t}, C)$ is higher than a given threshold $\theta$.

Citation

@misc{karkali2014temporalidf,
  title={Using temporal IDF for efficient novelty detection in text streams},
  author={Karkali et al. (2014)},
  year={2014},
  note={arXiv:1401.1456}
}

• arXiv: 1401.1456