Build Verdict Dataset

Build Verdict Dataset

Use this skill to create or refresh verdict_quality_dataset from human annotations in Weave, then publish it as a versioned Weave Dataset and pin the ref.

The coding agent queries annotated research_run_<i> calls, extracts candidate fields, maps the human verdict to a gold label, refines each row against the rubric, writes a local JSONL plus an audit report, publishes the dataset, and updates the pinned ref in evaluation_config.yaml.

This skill does not change the human annotation results themselves. It only transforms annotation evidence into a clean eval dataset and records provenance.

Before You Start

Follow AGENTS.md W&B Skills setup first. Use W&B Skills for Weave trace, feedback, annotation, and dataset access; this skill only defines the Discovery Forge dataset-building workflow.

Fetch all trace and feedback evidence live from Weave through W&B Skills. Do not use W&B MCP tools, and do not add discovery-forge query wrappers.

Default Project

• Entity: read from .env as WANDB_ENTITY (required; use your own W&B entity)
• Project: read from .env as WANDB_PROJECT (required; .env.example uses discovery-forge)
• API key: read from .env as WANDB_API_KEY (required)
• Annotation queue: research_annotation
• Root trace unit: one research_run_<i> / openai_agent_trace call per reviewed candidate
• Dataset name: verdict_quality_dataset
• Pinned ref: src/discovery_forge/evaluation/evaluation_config.yaml datasets.verdict_quality.ref
• Publish helper: discovery_forge.evaluation.datasets.publish_eval_dataset
• Labeling standard: src/discovery_forge/evaluation/verdict_dataset_rubric.md
• Local outputs: src/discovery_forge/evaluation/datasets/verdict_quality_dataset_clean_<YYYY-MM-DD>.jsonl and verdict_quality_dataset_audit_<YYYY-MM-DD>.md

Row Schema

Each dataset row must contain:

• id: stable slug for the candidate (kebab-case of the tool name)
• input_tool_name: candidate name
• input_candidate_url: candidate primary URL
• input_candidate_description: neutral candidate description (see hygiene rules)
• expected_scope_status: accepted or rejected (the only field the scorer reads)
• expected_issue_category: for rejects, one of out_of_scope, missing_url, duplicate_known_tool; otherwise null
• label_reason: human-readable rationale (not scored)
• annotation_source: {queue_name, call_id, quality_selector} provenance — copy verbatim from the source annotation, never invent or alter

Workflow

1. Identify the source scope: a set of research_run call IDs (from the user or a Weave UI link), a run day, or "all reviewed items in research_annotation". Prefer an explicit call-ID list or run day over "every historical annotation". Get call IDs from Weave, not from local files.
2. Use W&B Skills to fetch the annotated root calls and feedback from Weave.
3. For each call, separate human annotations (wandb.annotation.QualitySelector, wandb.annotation.QualityReviewer) from runnable scorer feedback. Use only research_annotation human annotations as the verdict seed; do not seed labels from runnable scorers.
4. Inspect one call first to learn the exact output shape, then extract the candidate name, primary URL, and a pre-verdict description plus the human reviewer's rationale.
5. Map the human QualitySelector to a draft gold label:
   • Good -> accepted
   • Bad -> rejected
   • Neutral -> ask the user how to handle it. Do not silently auto-assign. Present each neutral candidate with its QualityReviewer rationale and a recommended verdict (accepted with key_limitations, rejected, or drop), then let the user decide per row or give a blanket rule. Only fall back to a provisional label + needs_review flag if the user explicitly defers or says to proceed without them.
6. Refine each row against verdict_dataset_rubric.md (see Labeling and Hygiene). Record keep / relabel / drop / needs_review with a reason.
7. Write the clean JSONL and the audit report.
8. Publish the dataset and update datasets.verdict_quality.ref in src/discovery_forge/evaluation/evaluation_config.yaml.
9. Validate.

Evidence Selection

Use W&B Skills to fetch and inspect Weave evidence. This skill only defines which Discovery Forge evidence matters:

• Root trace unit: one research_run_<i> / openai_agent_trace call per reviewed candidate.
• Inspect only calls in the requested source scope.
• Keep only human annotations from research_annotation (QualitySelector, QualityReviewer) as verdict seeds.
• Do not seed labels from runnable scorer feedback.
• Deduplicate annotations by feedback ID.
• Inspect one representative call first to learn where candidate name, primary URL, description, reviewer rationale, and annotation payload live.

Labeling and Hygiene

Apply verdict_dataset_rubric.md. Key rules:

• Accept when the candidate itself runs a loop: task -> evaluation -> feedback/state -> revision of its own artifact.
• Reject lists/surveys/cookbooks/guides, GUI/computer-use frameworks, testing/evaluation-only gates, repository-automation hosts, memory-only components, generic frameworks, and weak-evidence candidates.
• Metadata vs scope: missing stars/license/dates/GitHub fetch is a profile limitation, not a scope reject. Do not encode metadata gaps as missing_url when a primary URL exists.
• Input hygiene: input_candidate_description must be neutral (no leaked verdict) but must carry enough factual evidence to support the verdict from the row input alone. A description with only Autonomy: X Domains: ... is too sparse — add the neutral artifact type and, for accepts, the concrete loop, verified against primary sources.
• Verify with primary sources before relabeling away from the human verdict. Record the source (repo, paper, docs) in the audit.

Provenance Rules

• Never change the human annotation result. Copy quality_selector and the originating call_id verbatim into annotation_source.
• When you relabel expected_scope_status away from the raw QualitySelector mapping, keep the original annotation_source and explain the rubric-based reason in the audit. The provenance must still point to the same human annotation.
• Do not invent annotations, call IDs, URLs, or dates.

Audit Report

Write verdict_quality_dataset_audit_<YYYY-MM-DD>.md with:

• Source scope (day / call IDs / queue) and the rubric path
• Row count and accepted/rejected distribution
• Per-row action table: id | action (keep/relabel/drop/needs_review) | expected | issue | reason
• Primary sources consulted for any relabel
• The published dataset ref after publishing

Publish And Pin

uv run python - <<'PY'
from pathlib import Path
from dotenv import load_dotenv
load_dotenv(".env")
import weave
from discovery_forge.observability import weave_project_path
from discovery_forge.evaluation.datasets import publish_eval_dataset, VERDICT_DATASET_NAME

weave.init(weave_project_path())
result = publish_eval_dataset(
    Path("src/discovery_forge/evaluation/datasets/verdict_quality_dataset_clean_<YYYY-MM-DD>.jsonl"),
    name=VERDICT_DATASET_NAME,
)
print(result)
PY

Then update datasets.verdict_quality.ref in src/discovery_forge/evaluation/evaluation_config.yaml to the new digest. Preserve older versions; publish a new one rather than overwriting.

Validation

uv run pytest tests/unit/test_eval_datasets.py tests/unit/test_researcher_evaluation.py -q
uv run python evaluate.py

Confirm the new ref resolves on the Weave server and that the eval links to verdict_quality_dataset (not an anonymous Dataset).

Apply Rules

• Build the dataset from research_annotation human annotations only; do not seed labels from runnable scorers.
• For Neutral annotations, ask the user how to handle each one before finalizing. Do not silently auto-assign neutral verdicts; only use a provisional label with a needs_review flag if the user explicitly defers.
• Do not change the human annotation results; only transform and refine into eval rows.
• Do not tune labels or descriptions to make a specific prompt score higher. Refine for correctness and clarity per the rubric, and record every change in the audit.
• Keep the scorer, evaluation runner, and registry unchanged in this workflow.

Report Back

Report:

• source scope (day / call IDs / queue) and number of annotations used
• row count and accepted/rejected distribution
• rows relabeled or dropped, with rubric reasons and primary sources
• published dataset ref and the updated evaluation_config.yaml entry
• validation results (tests, eval run, ref resolves)
• rows left as needs_review