# Conversation Disentanglement Eval

> Probes a model's ability to reconstruct reply relationships in multi-party, entangled text conversations. It requires identifying which message responds to which, handling simultaneous conversations, and distinguishing reply edges from system or directed messages. Use when the user wants to benchmark on IRC Conversation Disentanglement Corpus, or asks about evaluating this task. Reports F1 score for reply-edge prediction.

- Skill: `qhjqhj00/conversation-disentanglement-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/conversation-disentanglement-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/conversation-disentanglement-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/conversation-disentanglement-eval

---


# conversation-disentanglement-eval

> A Large-Scale Corpus for Conversation Disentanglement — Kummerfeld et al. (2018) (arXiv:1810.11118, 2018)

## What this evaluates

Probes a model's ability to reconstruct reply relationships in multi-party, entangled text conversations. It requires identifying which message responds to which, handling simultaneous conversations, and distinguishing reply edges from system or directed messages.

## Datasets

- **IRC Conversation Disentanglement Corpus** — total 77563; splits: test (-1)

## Metrics

- `F1 score for reply-edge prediction` **(primary)** — range: [0, 1]
  - Standard F1 score computed over predicted vs. gold reply edges in the conversation graph. Precision is the fraction of predicted edges that are correct; recall is the fraction of gold edges recovered. F1 is the harmonic mean of precision and recall.

## Input / output format

**Input**: A chronological sequence of messages from a shared IRC channel, including sender identifiers, timestamps, and message text.

**Output**: A directed graph where each node represents a message and each directed edge (A → B) indicates that message B is a reply to message A. Connected components define individual conversations.

## Scoring recipe

```python
def compute_f1_edges(pred_edges, gold_edges):
    pred_set = set(pred_edges)
    gold_set = set(gold_edges)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1
```

## Common pitfalls

- Only ~48% of messages contain explicit directed cues (e.g., @username), so models cannot rely solely on mentions.
- System messages start with '==' but are not all system messages, creating ambiguity in edge prediction.
- Users frequently participate in multiple conversations simultaneously, requiring context-aware disentanglement rather than simple clustering.

## Evidence (verbatim from paper)

> We label this data with a graph in which messages are nodes and edges indicate that one message is a response to another. Each connected component is a conversation.

## Citation

```bibtex
@misc{kummerfeld2018largescale,
  title={A Large-Scale Corpus for Conversation Disentanglement},
  author={Kummerfeld et al. (2018)},
  year={2018},
  note={arXiv:1810.11118}
}
```

- arXiv: 1810.11118