Scrapegraphai 100k Eval

scrapegraphai-100k-eval

ScrapeGraphAI-100k: A Large-Scale Dataset for LLM-Based Web Information Extraction — Brach et al. (2026) (arXiv:2602.15189, 2026)

What this evaluates

Evaluates LLM-based web information extraction by measuring structural validity (JSON parseability, schema compliance), key extraction accuracy (precision, recall, F1), and value extraction quality (type-aware exact match, BLEU) on real-world HTML-to-JSON tasks.

Datasets

• ScrapeGraphAI-100k — total 93695; splits: test (2714); HF scrapegraphai/scrapegraph-100k-finetuning; repo https://github.com/ScrapeGraphAI/scrapegraph-100k-paper

Metrics

• is_valid_json — range: [0, 1]
  • Boolean metric indicating whether the model output can be successfully parsed as valid JSON.
• is_schema_compliant — range: [0, 1]
  • Boolean metric checking if the parsed JSON output conforms to the target schema structure.
• Key precision — range: [0, 1]
  • Precision of flattened JSON keys using dot-notation paths with [*] wildcards for arrays.
• Key recall — range: [0, 1]
  • Recall of flattened JSON keys using dot-notation paths with [*] wildcards for arrays.
• Key F1 (primary) — range: [0, 1]
  • Harmonic mean of key precision and recall, measuring structural accuracy of extracted keys.
• Missing keys — range: other
  • Average number of gold keys absent from the model output per sample.
• Extra keys — range: other
  • Average number of hallucinated keys present in the model output but absent from gold per sample.
• Value score — range: [0, 1]
  • Type-aware metric averaging exact match for booleans/numbers, set equality for arrays, and sentence-level BLEU for strings.
• Overall BLEU — range: [0, 1]
  • Sentence-level BLEU score computed on the serialized JSON string of the model output.

Input / output format

Input: Raw HTML content (DOM structure) and a natural-language prompt specifying the extraction target, along with a JSON schema defining the desired output structure.

Output: A JSON object conforming to the target schema, containing extracted keys and values from the HTML.

Scoring recipe

def evaluate(pred_str, gold_str, schema):
    pred = json.loads(pred_str) if is_valid_json(pred_str) else None
    gold = json.loads(gold_str)
    valid = pred is not None
    schema_ok = check_schema(pred, schema) if valid else False
    p_keys = flatten_keys(pred)
    g_keys = flatten_keys(gold)
    tp = len(p_keys & g_keys)
    prec = tp / len(p_keys) if p_keys else 0
    rec = tp / len(g_keys) if g_keys else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0
    missing = len(g_keys - p_keys)
    extra = len(p_keys - g_keys)
    val_scores = [type_aware_match(pred[k], gold[k]) for k in g_keys if k in p_keys]
    value_score = mean(val_scores) if val_scores else 0
    overall_bleu = sentence_bleu(tokenize(pred_str), tokenize(gold_str))
    return valid, schema_ok, prec, rec, f1, missing, extra, value_score, overall_bleu

Common pitfalls

• Models may produce structurally valid JSON that perfectly matches the schema but contains hallucinated or imprecise values, creating a structural-semantic gap.
• Key matching uses dot-notation with [*] wildcards for arrays; failing to account for array wildcards will artificially deflate precision/recall.
• Evaluation excludes samples exceeding the 8192-token context limit, so results may not generalize to very long documents.

Evidence (verbatim from paper)

Structural validity metrics assess whether outputs are well-formed: is_valid_json checks JSON parseability, while is_schema_compliant verifies conformance to the target schema. Key extraction metrics (precision, recall, F1) measure structural accuracy by comparing flattened JSON keys using dot-notation paths with [] wildcards for arrays. Value extraction is captured by value_score, a type-aware metric averaging exact match for booleans/numbers, set equality for arrays, and sentence-level BLEU(Papineni et al., [2002])* for strings. Finally, overall_bleu on serialized JSON provides a holistic quality measure.

Citation

@misc{brach2026scrapegraphai100k,
  title={ScrapeGraphAI-100k: A Large-Scale Dataset for LLM-Based Web Information Extraction},
  author={Brach et al. (2026)},
  year={2026},
  note={arXiv:2602.15189}
}

• arXiv: 2602.15189