Dharmaocr Benchmark Eval

dharmaocr-benchmark-eval

DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselines — Cardoso et al. (2026) (arXiv:2604.14314, 2026)

What this evaluates

Evaluates structured OCR extraction fidelity and text degeneration rates on printed, handwritten, and legal documents. Measures how well models adhere to JSON schemas while minimizing pathological generation loops.

Datasets

• DharmaOCR-Benchmark — total ?; splits: test (-1)

Metrics

• Score (primary) — range: [0, 1]
  • Extraction fidelity score measuring how accurately the model outputs structured JSON matching the ground truth. Range: [0, 1].
• Degeneration rate (%) — range: percent
  • Percentage of evaluated pages where pathological generation loops or text degeneration occurs, degrading output quality.
• Time per page (s) — range: seconds
  • Average inference time required to process a single document page.
• Relative cost (%) — range: percent
  • Inference cost relative to a reference model (olmOCR 2-7B), calculated using hourly GPU costs and vLLM configuration.

Input / output format

Input: Document images (printed, handwritten, or legal) paired with a target JSON schema for structured extraction.

Output: Strictly formatted JSON object containing the extracted fields, validated against the provided schema.

Scoring recipe

def compute_score(pred_json, gold_json):
    # Measures extraction fidelity against ground truth structured output
    return exact_match_or_f1(pred_json, gold_json)

def compute_degeneration_rate(predictions):
    degenerate_count = 0
    for pred in predictions:
        if is_pathological_loop(pred.text) or violates_schema_strictly(pred):
            degenerate_count += 1
    return (degenerate_count / len(predictions)) * 100

Common pitfalls

• Cost comparisons mix local GPU hourly rates with commercial API public prices, potentially disadvantaging commercial providers.
• Degeneration rate specifically targets pathological generation loops, not just standard OCR errors.
• Quantization (AWQ) impacts both latency and cost, requiring careful tracking of quality trade-offs.

Evidence (verbatim from paper)

Table 1 summarizes the evaluation results for the models explored, including proprietary models, open-source baselines models, and variants trained. Since the cost of non-commercial models depends on the execution infrastructure, a relative cost is expressed as a percentage with respect to a reference model. OlmOCR 2-7B, which achieved the most consistent benchmark outcome (high score and low text degeneration rate), was chosen as the reference model.

Citation

@misc{cardoso2026dharmaocr,
  title={DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselines},
  author={Cardoso et al. (2026)},
  year={2026},
  note={arXiv:2604.14314}
}

• arXiv: 2604.14314