02 Evaluation Datasets

Evaluation Dataset Creation

Patterns for building, validating, versioning, and logging evaluation datasets for GenAI agents. Complements SDLC Step 4 (evaluation runs and scorers) by focusing on data shape, UC persistence, and dataset lifecycle, grounded in Databricks MLflow GenAI eval monitor documentation.

Upstream Lineage

This skill extends Databricks Agent Skills' databricks-mlflow-evaluation skill for evaluation dataset construction, production trace-to-dataset workflows, and expectation schema guidance. If local dataset guidance is insufficient or MLflow evaluation dataset APIs drift, consult the upstream skill first, then adapt its patterns to this workshop's canonical row fields and SDLC artifact contracts.

When to Use

• Defining benchmark suites for agent quality gates
• Persisting rows to Unity Catalog Delta tables behind mlflow.genai.datasets
• Loading and merging records without schema surprises across jobs or tasks
• Validating rows before mlflow.genai.evaluate() so bad data never skews metrics
• Balancing splits (train / held_out) and provenance (curated, synthetic, human_labeled)

Official data format (mlflow.genai.evaluate)

Per the eval harness parameters, each record aligns with EvaluationDataset schema. Either inputs + outputs or trace is required; you cannot pass both.

Direct evaluation: rows typically have inputs and optional expectations only.

Answer sheet: rows have inputs + outputs, or a single trace, plus optional expectations.

EvaluationDataset vs DataFrame / list-of-dicts

Databricks recommends mlflow.genai.datasets.EvaluationDataset when available: it enforces schema validation and improves lineage tracking. Raw pandas DataFrame, list of dicts, or Spark DataFrame are accepted if they match the same column semantics. See Building MLflow evaluation datasets.

Evaluation record schema (transferable)

mlflow.genai.evaluate() unpacks each row's inputs dict as keyword arguments to predict_fn. Every key in inputs must match the predictor's signature (use **kwargs only for optional or closure-fed values).

Minimal shape (docs and quick tests)

# Chat / generic agent
{"inputs": {"question": "What is X?"}, "expectations": {"reference_answer": "..."}}

# RAG agent — pass what your app and scorers need
{
    "inputs": {"query": "...", "retrieved_context": "..."},
    "expectations": {"citations": ["doc:page"], "answer": "..."},
}

# SQL / tool agent
{
    "inputs": {"question": "Total cost by region?"},
    "expectations": {"expected_sql": "SELECT ...", "expected_tables": ["db.schema.t"]},
}

Production-oriented shapes (examples)

Use only fields your predict_fn and scorers actually read. Typical additions:

If scorers read ground truth from expectations, mirror any critical label from inputs into expectations when your judges expect a fixed key (for example SQL string under both inputs["expected_sql"] and expectations["expected_response"]).

Canonical evaluation-dataset fields (normative)

Every row in a benchmark dataset MUST carry the following canonical fields, regardless of agent shape. These names are the contract between dataset producers, runners, and scorers — do not invent synonyms.

eval_dataset_required_fields:
  - row_id              # stable per-row identifier (used as merge key, dedup key, regression-tracking key)
  - request             # the user-facing input (mirrored under inputs.<key> for predict_fn)
  - expected_response   # canonical ground truth consumed by the Correctness scorer
  - expected_signal     # SECONDARY classification field (e.g. intent, severity, topic) — never silently mirrored into expected_response
  - bucket              # coverage bucket (e.g. "aggregation", "ranking", "edge_case", "permission_denied")
  - journey_id          # which user journey this row exercises (links to ui.user_journeys)
  - split               # train | held_out | regression | gold
  - provenance          # curated | synthetic | auto_corrected | issue_failing_trace | labeling_session_merge

Hard rules:

• expected_response is the ONLY field the Correctness scorer reads as ground truth. Populate it directly with the answer string (or canonical SQL) the agent should produce.
• expected_signal is allowed only as a secondary classification field (e.g. expected_signal = "policy_refusal" for a guardrail label). The runner MUST NOT silently mirror expected_signal into expected_response. If a scorer needs the signal as ground truth, write it under expected_response explicitly with intent.
• row_id is required for merge_records upserts and for tagging regressions across runs. Use a stable hash of (request, journey_id) if you do not have a natural key.
• bucket, journey_id, and split together support coverage gates (see below).

Coverage gates (normative thresholds)

Datasets that do not meet the following minima MUST fail pre-evaluation validation:

min_rows: 40
per_bucket_min_rows: 1
per_journey_min_rows: 1
expectations_schema_complete: true              # all required fields present + non-empty for split != "regression"
eval_dataset_canonical_source: enum             # uc_table | local_json | labeling_session_merge

expectations_schema_complete means: for every row where split != "regression", all canonical fields above are present and expected_response is non-empty. Regression rows may carry expected_response = null only when the row is gated on a scorer threshold (see references/benchmark-generation.md §11).

eval_dataset_canonical_source records where the row set came from. The runner reads this once at startup and refuses to evaluate if the value is unknown — this prevents silent mixing of incompatible sources.

Creating and persisting datasets (UC)

Table naming: use a stable Unity Catalog identifier, e.g. {catalog}.{schema}.{app_name}_benchmarks (replace with your catalog, schema, and app slug).

Tabular construction (pandas)

import pandas as pd

def rows_to_eval_df(questions, expectations=None):
    records = []
    for i, q in enumerate(questions):
        r = {"inputs": {"question": q}}
        if expectations and i < len(expectations):
            r["expectations"] = expectations[i]
        records.append(r)
    return pd.DataFrame(records)

get_dataset + merge_records (SDK)

Aligns with Create a dataset using the SDK:

import mlflow.genai.datasets

uc_table = f"{catalog}.{schema}.{app_name}_benchmarks"
try:
    eval_dataset = mlflow.genai.datasets.get_dataset(name=uc_table)
except Exception:
    eval_dataset = mlflow.genai.datasets.create_dataset(name=uc_table)

records = [...]  # list of dicts: inputs / outputs / expectations / trace per rules above
eval_dataset.merge_records(records)

After merges, eval_dataset.to_df() (or your validated in-memory frame) is the source of truth for row counts and deduplication before evaluation.

Loading for evaluation

1. Preferred: materialize from the EvaluationDataset or the DataFrame you already validated in memory.
2. Spark / Delta: if you read {catalog}.{schema}.{app_name}_benchmarks directly, use REFRESH TABLE (or equivalent) when another task may have altered the table in the same run, then parse JSON columns if inputs / expectations are stored as strings.

Generic read pattern:

def load_eval_rows_spark(spark, full_table_name: str):
    spark.sql(f"REFRESH TABLE {full_table_name}")
    return spark.table(full_table_name)

Best practice: evaluate from deduped in-memory data, not the raw UC table

merge_records upserts by record identity; if the same logical example is merged with different IDs, the Delta table can accumulate stale duplicates. Downstream, reading the table without the same dedupe logic as merge can inflate row counts and distort metrics.

Do: dedupe in memory (for example by stable business key: normalized question, session+turn, or hash of inputs) and pass that DataFrame or list to mlflow.genai.evaluate().

Don't: assume the UC table is duplicate-free or that merge_records removed older variants unless your merge keys guarantee it.

Generating benchmarks (generic)

Common sources (see also Data sources for evaluation datasets):

• Production traces — sample via mlflow.search_traces(), filter for quality or failure modes, merge into the dataset after review.
• SME-curated — subject-matter experts label inputs and expectations; best for gold sets and regressions.
• LLM-generated — expand coverage quickly; always validate (schema, permissions, tool contracts) before merge or evaluate.

Validate structure, tool outputs, and permissions before calling mlflow.genai.evaluate() so invalid rows never enter scored runs.

Load references/synthetic-eval-generation.md if you need a full recipe for generating a starter eval set from production traces (cluster traces by intent → LLM-generate paraphrases → expectations harvested from SME-labeled traces → validate → merge).

Load references/benchmark-generation.md if you need end-to-end LLM benchmark generation, Genie Q&A extraction, SQL validation, and issue-focused subsets from failing traces.

Dataset versioning and lineage

• Storage: UC Delta tables behind mlflow.genai.datasets (name like {catalog}.{schema}.{app_name}_benchmarks).
• Runs: log dataset identity and row counts on the evaluation run (see Evaluation runs).
• Deduping: enforce stable keys at merge time and again before evaluate.
• Balance: track counts per category or split when mixing curated and synthetic data.

Common mistakes

DO / DON'T

DO — populate every canonical field; put Correctness ground truth under expected_response:

record = {
    "row_id": "billing_aggregation_001",
    "inputs": {"request": "Total by region?", "expected_sql": sql},
    "expectations": {
        "expected_response": sql,           # canonical ground truth for Correctness
        "expected_signal": "aggregation",   # SECONDARY classification — never mirrored into expected_response
        "bucket": "aggregation",
        "journey_id": "cost_analysis",
        "split": "train",
        "provenance": "curated",
    },
}

DON'T — silently mirror expected_signal into expected_response (Correctness will score against the wrong target):

# WRONG — runner copying expected_signal into expected_response
record["expectations"]["expected_response"] = record["expectations"]["expected_signal"]

DON'T — empty expectations when judges need ground truth:

record = {"inputs": {"request": "...", "expected_sql": sql}, "expectations": {}}

DO — get_dataset with fallback to create_dataset, then merge_records.

DON'T — always create_dataset without checking existence (duplicates / errors).

DO — validate rows (schema, tools, SQL, permissions) before evaluate.

DON'T — trust synthetic outputs without checks.

DO — log dataset name and row_count (and optional validation summaries) on the MLflow run.

DON'T — run evaluate() with no record of which dataset version was used.

Validation checklist

Schema and predictor

• Each row follows harness rules: inputs+outputs or trace, not both
• inputs keys match predict_fn for direct evaluation
• expectations keys match what scorers read
• All canonical fields present: row_id, request, expected_response, expected_signal, bucket, journey_id, split, provenance
• Correctness ground truth lives under expected_response only — expected_signal is NOT mirrored into it

Coverage gates

• min_rows: 40 satisfied
• per_bucket_min_rows: 1 satisfied for every declared bucket
• per_journey_min_rows: 1 satisfied for every ui.user_journeys entry
• expectations_schema_complete: true (all canonical fields present, expected_response non-empty for split != "regression")
• eval_dataset_canonical_source set to one of uc_table | local_json | labeling_session_merge

UC and MLflow

• Dataset name {catalog}.{schema}.{app_name}_benchmarks consistent across writers and loaders
• After concurrent DDL/DML, refresh reads in multi-task jobs
• Evaluation uses deduped data aligned with merge semantics

Content

• Tool / SQL / RAG outputs validated where applicable
• Provenance and split fields populated for traceability
• Regression rows augment, not replace, baseline rows (see references/benchmark-generation.md §11)

Scripts

References

Official documentation

• Building MLflow evaluation datasets
• Evaluate GenAI during development (eval harness)
• Evaluation runs
• Evaluation dataset reference (UI + schema)
• MLflow genai.evaluate

Related skills

• SDLC Step 3 (03-scorers-and-judges) — scorers, judges, predict_fn contract
• SDLC Step 1 — prompts used during synthetic or judged workflows
• SDLC Step 4 — wiring predict_fn and evaluation execution

Local reference files

Version history