Webmainbench Eval

webmainbench-eval

Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM — Liu et al. (2025) (arXiv:2511.23119, 2025)

What this evaluates

Evaluates a model's ability to extract main content from HTML web pages by classifying semantic blocks and generating clean text or Markdown. It probes robustness across varying difficulty levels and rich content types such as tables, code, and equations.

Datasets

• WebMainBench — total 7800; splits: test (7800)
• WCEB — total ?; splits: test (-1)

Metrics

• ROUGE-N F1 (primary) — range: [0, 1]
  • Standard ROUGE-N F1 score computed between the model's extracted text/markdown and the ground truth text. The paper reports the F1 measure for N-gram overlaps.

Input / output format

Input: HTML document (raw or simplified), with a maximum context length of 32,000 tokens for the standard model.

Output: JSON classification of semantic blocks, which is subsequently converted to Markdown or plain text for evaluation. Inputs exceeding the context limit receive a score of 0.

Scoring recipe

def compute_metric(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        if pred is None or len(pred.strip()) == 0:
            scores.append(0.0)
        else:
            pred_text = convert_html_to_text(pred)
            gold_text = convert_html_to_text(gold)
            scores.append(rouge_f1(pred_text, gold_text))
    return sum(scores) / len(scores)

Common pitfalls

• Inputs exceeding the 32k token context window are assigned a score of 0 for the standard model, which can artificially lower overall scores if not filtered or handled via fallback.
• Baseline tools output diverse formats (HTML+MD, MD, TEXT); evaluation requires consistent conversion to a common text format before computing ROUGE-N F1 to ensure fairness.

Evidence (verbatim from paper)

Table 2: Performance comparison on MainWebBench (ROUGE-N F1). Methods are categorized by Mode: Html+MD denotes tools outputting intermediate HTML converted to Markdown, while MD and TEXT denote native Markdown and Plain Text outputs, respectively.

Citation

@misc{liu2025dripper,
  title={Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2511.23119}
}

• arXiv: 2511.23119