Dynamic Topic Quality Eval

dynamic-topic-quality-eval

Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion — Xiaobao Wu et al. (2024) (arXiv:2405.17957, 2024)

What this evaluates

Evaluates dynamic topic models by measuring topic coherence and diversity across chronological time slices, and assesses the utility of learned document-topic distributions via downstream text classification and clustering tasks.

Datasets

• NeurIPS — total ?; splits: test (-1); repo https://www.kaggle.com/datasets/benhamner/nips-papers
• ACL — total ?; splits: test (-1)
• UN — total ?; splits: test (-1); repo https://www.kaggle.com/unitednations/un-general-debates
• NYT — total ?; splits: test (-1); HF Matthewww/nyt_news
• WHO — total ?; splits: test (-1)

Metrics

• Topic Coherence (TC) (primary) — range: [0, 1]
  • Measures the coherence of top words in a topic using the $C_V$ metric. It computes word occurrence probabilities using the documents of the specific time slice as a reference corpus. Higher scores indicate stronger association with the slice and fewer unassociated topics.
• Topic Diversity (TD) — range: [0, 1]
  • Computes the proportion of top words in a topic that occur exactly once within the documents of that specific time slice. Higher scores indicate topics are more distinct from each other, verifying the absence of repetitive topics.
• Accuracy — range: [0, 1]
  • Standard classification accuracy calculated by training an SVM classifier on learned document-topic distributions as features and predicting document categories.
• F1 — range: [0, 1]
  • Macro-averaged F1 score from the same SVM classification setup as Accuracy.
• Purity — range: [0, 1]
  • Clustering purity metric computed by assigning documents to their most significant topic and comparing against ground-truth categories.
• NMI — range: [0, 1]
  • Normalized Mutual Information between topic-based cluster assignments and ground-truth document categories.

Input / output format

Input: Collections of documents partitioned into chronological time slices. Each instance consists of a time slice containing multiple documents.

Output: Topic-word distributions (top words per topic) and document-topic distributions (probability of each topic per document) for each time slice.

Scoring recipe

def evaluate_model(model, datasets, K=50):
    avg_tc, avg_td = [], []
    for dataset in datasets:
        for slice_t in dataset.slices:
            topics = model.get_topics(slice_t, K)
            for topic in topics:
                top_words = topic.top_n_words
                # TC: C_V coherence using slice_t docs as reference
                avg_tc.append(c_v_score(top_words, slice_t.docs))
                # TD: proportion of top words occurring exactly once in slice_t
                counts = count_word_freq(top_words, slice_t.docs)
                avg_td.append(sum(1 for w in top_words if counts[w] == 1) / len(top_words))
    return mean(avg_tc), mean(avg_td)

# Downstream: Train SVM on doc-topic distributions for Acc/F1.
# Use top topics as cluster labels for Purity/NMI.

Common pitfalls

• TC is computed using the time slice's own documents as the reference corpus, not the entire dataset, to measure slice association.
• TD counts top words that occur exactly once within the specific time slice, not globally across all slices.
• Downstream classification and clustering use the learned document-topic distributions as features or cluster assignments, not raw text or topic words directly.

Evidence (verbatim from paper)

To compare dynamic topic models, we evaluate the quality of topics from two aspects following Dieng et al. (2019): (i) Topic Coherence(TC), referring to the coherence of top words in a topic. We employ the popular $C_{V}$ as the coherence metric Röder et al. (2015)... To measure the association between a topic and its time slice as well, we compute its TC with the documents of its slice as a reference corpus... (ii) Topic Diversity(TD), referring to difference between discovered topics Dieng et al. (2020). For a topic at a slice, we compute the proportion of its top words that only occur once and also exist at that slice.

Citation

@misc{wu2024modeling,
  title={Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion},
  author={Xiaobao Wu et al. (2024)},
  year={2024},
  note={arXiv:2405.17957}
}

• arXiv: 2405.17957