# Annotation Improvement > Guides coding agents through Discovery Forge prompt improvement after selecting the `wandb-primary` skill. Use `wandb-primary` to fetch W&B Weave research_run traces, human annotations, runnable feedback, and evaluations, then use this workflow when improving researcher.md from annotation queues, reviewed research traces, or human feedback. - Skill: `wandb/annotation-improvement` (Agent Skill) - Install (CLI): `npx skillmds@latest add wandb/annotation-improvement` - Raw SKILL.md: https://api.skillmd.com/api/skills/wandb/annotation-improvement/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Product & Planning - Author: wandb (https://skillmd.com/u/wandb) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/wandb/annotation-improvement --- # Annotation Improvement Use this skill when improving `src/discovery_forge/agents/researcher.md` from live Weave annotation evidence fetched with the `wandb-primary` skill. The coding agent fetches evidence, writes a plan, edits the prompt, publishes the prompt, and validates the change. ## Before You Start Follow `AGENTS.md` setup first, then select and use the installed `wandb-primary` skill for Weave trace, feedback, annotation, prompt, and evaluation access. This skill only defines the Discovery Forge prompt-improvement workflow. Fetch all trace and feedback evidence live from Weave through `wandb-primary` skill guidance, including its Weave SDK / W&B API patterns. Do not use W&B MCP tools, and do not add discovery-forge query wrappers. ## Default Project - Entity: read from `.env` as `WANDB_ENTITY` (required; use your own W&B entity) - Project: read from `.env` as `WANDB_PROJECT` (required; `.env.example` uses `discovery-forge`) - API key: read from `.env` as `WANDB_API_KEY` (required) - Prompt file: `src/discovery_forge/agents/researcher.md` - Evidence source: Weave traces and feedback fetched live via the `wandb-primary` skill - Improvement history: `src/discovery_forge/agents/improve_history//plan.md` and `src/discovery_forge/agents/improve_history//applied.md` - Root trace unit: one `research_run_` / `openai_agent_trace` call per discovered tool ## Evidence Workflow 1. Identify the target from what the user gives you: explicit Weave call IDs, a Weave Evaluation link / eval call ID, a run day, or a run ID. 2. Use the `wandb-primary` skill to fetch the target root calls and feedback **from Weave**. If you only have a day or run ID, use `wandb-primary` Weave SDK guidance to identify the matching root calls for that scope. 3. Include feedback evidence for the same root calls. 4. Read these fields per root call: `id`, `display_name`, `output`, `attributes`, `summary`, `feedback`. 5. Separate feedback into: - human annotations: `wandb.annotation.QualitySelector`, `wandb.annotation.QualityReviewer` - runnable scorer feedback: `Researcher-quality-check`, `Researcher-category-check`, `Quality-classifiers` 6. Prefer human annotations for final quality judgment, but use runnable scorer feedback for concrete evidence failures such as missing sources, unsupported claims, placeholder URLs, wrong category, or hallucination risk. 7. Read the current `researcher.md`. 8. Write `src/discovery_forge/agents/improve_history//plan.md` before editing. If the file already exists for that day, overwrite it with the latest plan. ### Evidence Selection Use the `wandb-primary` skill to fetch and inspect Weave evidence. This skill only defines which Discovery Forge evidence matters: - Root trace unit: one root call per discovered tool, display name `research_run_` (or `openai_agent_trace`). - For explicit Weave call IDs, inspect exactly those calls. - For a day/run target, inspect the matching `research_run_*` root calls for that scope. - For an evaluation target, inspect the parent evaluation call and its child rows. - Read only the evidence needed for analysis: call identity, display name, output, attributes, summary, and feedback. Current ResearcherAgent outputs include reviewer payload fields (`profile_review_markdown`, `verdict`, `tool_name`, `primary_url`, `summary`, `search`, `run`) and may also include eval-compatible fields (`scope_status`, `verdict_reason`, `final_output`, `profile`). - Separate human annotations from runnable scorer feedback in the plan. ### Date Scope - Prefer run day from the Weave call output or attributes over annotation creation date. - Include feedback only when the annotated call belongs to the requested day, run ID, or stage. - Deduplicate feedback by feedback ID. - Do not treat all historical annotations from a shared queue as current evidence. ### Offline Evaluation Evidence If the user provides a Weave Evaluation link or eval call ID: 1. Inspect that exact evaluation before rerunning anything. 2. Use the `wandb-primary` skill to inspect the parent evaluation call first. 3. Use the `wandb-primary` skill to inspect the evaluation child rows. 4. Read only the fields you need first: inputs, output, scorer outputs, display name, status. 5. Limit initial child rows to failed scorer rows or a small sample, then broaden only if needed. Evaluation datasets, audit sidecars, and scorers are read-only evidence. Do not change dataset rows, labels, dedup logic, scorer logic, or evaluation runner behavior to improve scores. ## Plan Format Use this structure: ```markdown # Skill-Based Prompt Improvement Plan for ## Source - Weave traces and feedback fetched via the `wandb-primary` skill - Human annotations inspected - Runnable scorer feedback inspected - Current `researcher.md` inspected ## Feedback Summary - ## Proposed Prompt Change - ## Applied Files - `src/discovery_forge/agents/researcher.md` ## Out of Scope - ``` ## Apply Rules - Prompt-only changes may edit only `src/discovery_forge/agents/researcher.md`. - Do not change evaluation datasets or scorers to make results look better. - Do not create fallback behavior. - Keep scope verdict separate from metadata completeness. - If primary sources prove the loop but metadata is incomplete, record the weakness in `key_limitations` instead of rejecting solely for metadata incompleteness. - Never invent placeholder URLs, paper IDs, arXiv IDs, citations, dates, or docs links. Use `unknown` or `null` when unverified. - If feedback includes explicit candidate names, URLs, or phrases ("search for X", "missed X", "exclude Y"), preserve the concrete examples in the plan. Use them as query examples or policy examples, not as hardcoded required outputs. - If `expected_scope_status == accepted` and the agent rejects due to missing metadata, keep scope verdict separate from metadata completeness. - If `expected_scope_status == rejected` and the agent accepts, treat it as a likely scope-policy issue unless the dataset row itself appears wrong. - If a dataset row, annotation label, or scorer appears wrong, report it as dataset/scorer maintenance rather than editing those artifacts in this workflow. After editing, write `src/discovery_forge/agents/improve_history//applied.md`. If the file already exists for that day, overwrite it with the latest applied summary. ## Publish Prompt After a prompt edit, publish the updated prompt using the project helper: ```bash uv run python - <<'PY' from discovery_forge.observability import init_observability from discovery_forge.tools.prompts import publish_instruction_prompts, prompt_hashes, prompt_refs init_observability(day_id="-skill-improve") versions = publish_instruction_prompts(max_tools=5) print(prompt_refs(versions)) print(prompt_hashes(versions)) PY ``` Record the prompt ref and prompt hash in `src/discovery_forge/agents/improve_history//applied.md`. ## Validation Run focused validation: ```bash uv run pytest tests/unit/test_researcher.py -q ``` If the change touches code outside `researcher.md`, add or update focused tests and run the relevant unit tests. ## Report Back Report: - day/run inspected - number of root traces queried - feedback types inspected - prompt changes made - Weave prompt ref/hash published - tests run - residual risks or items needing manual review