Liveweb Ie Eval

liveweb-ie-eval

LiveWeb-IE: A Benchmark For Online Web Information Extraction — Seungbin Yang et al. (2026) (arXiv:2603.13773, 2026)

What this evaluates

Evaluates web information extraction systems on live, dynamically evolving websites by testing their ability to identify target attributes and extract corresponding values from natural language queries. It probes the robustness of extraction pipelines against real-time web dynamics and complex layouts that break static HTML parsing.

Datasets

• LiveWeb-IE — total ?; splits: test (-1); repo https://github.com/sbY99/LiveWeb-IE

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall calculated over extracted values for each aligned attribute. Precision is the fraction of extracted values that are correct, and recall is the fraction of ground-truth values that are correctly extracted.

Input / output format

Input: Natural language query specifying target attributes, plus the HTML source or rendered view of a live web page.

Output: A set of extracted attribute-value pairs corresponding to the query.

Scoring recipe

# Align inferred attributes to ground-truth using GPT-4o
aligned_preds = llm_align(predictions, ground_truth_attributes)
# For each aligned attribute, compare extracted values to ground-truth values
tp, fp, fn = 0, 0, 0
for attr in aligned_preds:
    pred_vals = set(extracted_values[attr])
    gt_vals = set(ground_truth_values[attr])
    tp += len(pred_vals & gt_vals)
    fp += len(pred_vals - gt_vals)
    fn += len(gt_vals - pred_vals)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Attribute names may vary semantically between the system output and ground truth, requiring LLM-based alignment before value comparison.
• Evaluation is conducted in a zero-shot setting due to context length limits, so models cannot be fine-tuned or prompted with few-shot examples.
• Baselines operate on static HTML snapshots, while the benchmark tests live web dynamics, making direct comparison sensitive to rendering differences.

Evidence (verbatim from paper)

We evaluate how effectively a WIE system identifies the target attributes from a natural language query and extracts their corresponding values. First, we determine if the WIE system correctly identifies the target attributes. To account for the semantic flexibility of natural language, we employ an LLM-based alignment strategy, using GPT-4o to match the inferred attributes with their ground-truth counterparts. Inferred attributes that do not align are considered an extraction failure. Subsequently, for each aligned attribute, we compare the extracted values against the ground-truth values. In line with existing evaluation schemes for WIE, we measure performance using precision, recall, and F1 score.

Citation

@misc{yang2026livewebie,
  title={LiveWeb-IE: A Benchmark For Online Web Information Extraction},
  author={Seungbin Yang et al. (2026)},
  year={2026},
  note={arXiv:2603.13773}
}

• arXiv: 2603.13773