Construct Eval

construct-eval

Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction — Goh et al. (2026) (arXiv:2603.18014, 2026)

What this evaluates

Evaluates the trustworthiness and accuracy of LLM-generated structured outputs (JSON) against a ground truth or expected schema. It probes the model's ability to detect per-field and per-document errors in data extraction tasks without requiring labeled data.

Datasets

• Four real-world datasets (unspecified in excerpt) — total ?; splits: test (-1); repo https://github.com/cleanlab/structured-output-benchmark

Metrics

• rating (primary) — range: [1, 10]
  • An LLM judge evaluates the generator's structured output against the user request and schema, outputting a score from 1 to 10. Higher scores indicate greater trustworthiness and accuracy.

Input / output format

Input: User request (including document to process), desired structured output schema, and the generator model's structured output response.

Output: Per-document: a short explanation followed by a rating in format 'Rating: [[x]]' (1-10). Per-field: a JSON object mapping each field name to an object containing 'explanation' (str) and 'rating' (int, 1-10).

Scoring recipe

def score_trustworthiness(input_text, schema, generated_output):
    # Per-document scoring
    doc_prompt = build_prompt(input_text, schema, generated_output)
    doc_response = call_llm_judge(doc_prompt)
    doc_score = extract_int(doc_response, pattern=r'\[\[(\d+)\]\]')
    
    # Per-field scoring
    field_scores = {}
    for field_name in schema.keys():
        field_prompt = build_prompt(input_text, schema, generated_output, ignore_others=[field_name])
        field_response = call_llm_judge(field_prompt)
        field_scores[field_name] = extract_int(field_response, pattern=r'\[\[(\d+)\]\]')
    return doc_score, field_scores

Common pitfalls

• The LLM-as-Judge prompt is adapted from Zheng et al. (2023) and may inherit calibration biases or positional biases common to LLM judges.
• Per-field scoring requires precise schema parsing to map judge outputs back to original fields, especially for nested or dynamically generated JSON structures.
• Ratings are subjective 1-10 scores and may not linearly correlate with exact ground-truth accuracy without thresholding or calibration.

Evidence (verbatim from paper)

After providing your explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[rating]", for example: "Rating: [[5]]".

Citation

@misc{goh2026construct,
  title={Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction},
  author={Goh et al. (2026)},
  year={2026},
  note={arXiv:2603.18014}
}

• arXiv: 2603.18014