A2ui Add Eval Datapoint

Adding and verifying A2UI evaluation data points

Use this skill when adding new data points or datasets to the A2UI evaluation framework in eval/.

References

For full architecture and setup details, consult:

• eval/CONTRIBUTING_USE_CASES.md: The authoritative contributor guide for evaluation use cases, containing field definitions, context rules, and unencrypted multi-turn examples.
• eval/README.md: Evaluation quickstart, Transcrypt setup, and CLI flags.
• eval/DESIGN.md: Architecture, multi-stage scorers, and encryption-at-rest design.
• eval/datasets/dataset_schema.json: Formal JSON Schema defining required fields and structural rules for evaluation data points.

Step-by-step workflow

1. Unlock Transcrypt

Datasets in eval/datasets/ are encrypted at rest. Unlock Transcrypt locally before authoring (ask an A2UI team member for the password):

cd eval
bin/transcrypt -y -c aes-256-cbc -p <PASSWORD>

2. Author the data point

Create or update a dataset file in eval/datasets/<dataset_name>.yaml (e.g. eval/datasets/my_dataset.yaml).

Every data point must conform to the JSON Schema in eval/datasets/dataset_schema.json and follow the authoring guidelines in eval/CONTRIBUTING_USE_CASES.md.

For a complete unencrypted reference example of a multi-turn conversation with system instructions, tool calls (including unrelated background tool calls), tool responses, and a judging target rubric, inspect eval/examples/example_eval_case.json.

Key authoring requirements from eval/CONTRIBUTING_USE_CASES.md:

• Include the full multi-turn conversation history (messages), including assistant function calls (tool_calls) and tool responses (role: tool). Do not use single-turn prompts.
• Write the target as a qualitative judging rubric for the LLM-as-a-judge (what UI components must appear, data binding rules, and what errors to penalize). Do not include a hardcoded JSON string, as that ties the evaluation to a particular inference format.

3. Validate schema compliance

Verify that all data points satisfy eval/datasets/dataset_schema.json:

cd eval
uv run python -m pytest tests/test_dataset.py

4. Run an evaluation on the new data point

Always execute an evaluation run on the newly added dataset to verify model inference and scoring:

cd eval
# Quick validation on gemini-3.1-flash-lite
uv run main.py --dataset my_dataset --sanity

# Full evaluation check
uv run main.py --dataset my_dataset

View the interactive traces and judging rationales:

uv run inspect view start

5. Verify encryption and commit

When staging changes, Git applies the Transcrypt clean filter. Confirm that the staged file is encrypted ciphertext before committing:

# Stage the new dataset file
git add eval/datasets/my_dataset.yaml

# Verify staged content is encrypted ciphertext (not plaintext)
git diff --cached eval/datasets/my_dataset.yaml

# Commit the encrypted data point
git commit -m "feat(eval): add my_dataset evaluation data points"