Corpus Technical Validation Eval

corpus-technical-validation-eval

Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining — Amiri et al. (2026) (arXiv:2604.12498, 2026)

What this evaluates

Evaluates the structural integrity, metadata completeness, and text quality of a legally screened chemistry corpus derived from S2ORC. It verifies schema compliance, metadata field availability, subfield label validity, chunking consistency, and embedding reproducibility against predefined thresholds.

Datasets

• Lit2Vec Chemistry Corpus — total 582683; splits: full (582683)

Metrics

• schema_pass_rate (primary) — range: percent
  • Percentage of records passing all schema and structural checks, including JSON schema compliance, field typing, and embedding format validity.
• metadata_pass_rate — range: percent
  • Percentage of records with complete bibliographic fields (title, authors, venue, year, DOI, license) without warnings.
• text_quality_pass_rate — range: percent
  • Percentage of records passing length limits, Unicode validity, ASCII/whitespace ratios, and abstract–full-text alignment checks.
• chunking_pass_rate — range: percent
  • Percentage of documents passing token-size limits, Unicode integrity, and one-to-one paragraph–embedding mapping checks.
• ROUGE-1 — range: [0, 1]
  • Recall of ROUGE-1 unigrams between abstracts and full-text introductions to measure lexical alignment.

Input / output format

Input: JSON records containing full-text articles, abstracts, metadata, and paragraph-level chunks from the S2ORC-derived corpus.

Output: Validation status per record (pass/warn/fail) and aggregate pass rates per validation area. For text alignment, ROUGE-1 recall scores per record.

Scoring recipe

def evaluate_corpus(records):
    n = len(records)
    schema_pass = sum(1 for r in records if r.schema_valid)
    metadata_pass = sum(1 for r in records if r.metadata_complete)
    text_pass = sum(1 for r in records if r.text_quality_valid)
    chunk_pass = sum(1 for r in records if r.chunking_valid)
    rouge_scores = [rouge1_recall(r.abstract, r.full_text_intro) for r in records]
    return {
        'schema_pass_rate': schema_pass / n,
        'metadata_pass_rate': metadata_pass / n,
        'text_quality_pass_rate': text_pass / n,
        'chunking_pass_rate': chunk_pass / n,
        'ROUGE-1': rouge_scores
    }

Common pitfalls

• Missing or short abstracts (<100 chars) are excluded from embedding/subfield enrichment by design, triggering 'non-complete' schema flags that are not model failures.
• License warnings often reflect upstream S2ORC metadata gaps rather than true licensing conflicts, as the workflow enriches with OpenAlex/Crossref/Unpaywall.
• Short final chunks (<80-100 tokens) trigger warnings but are expected due to paragraph boundaries, not chunking errors.

Evidence (verbatim from paper)

Alignment analysis (Fig. 7) showed that approximately 355,000 records (72%) achieved strong lexical alignment between abstracts and the introductory portions of their corresponding full texts (ROUGE-1≥0.95). Fewer than 10% of records had low alignment (<0.5), typically in combination with other quality flags. Most records (398,064) had no content-quality flags.

Citation

@misc{amiri2026lit2vec,
  title={Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining},
  author={Amiri et al. (2026)},
  year={2026},
  note={arXiv:2604.12498}
}

• arXiv: 2604.12498