Wandb Primary

W&B Primary Skill

Environment defaults

• Python: run scripts with the Python environment available to your coding agent. Install missing optional packages only when needed.
• Credentials: use WANDB_API_KEY, WANDB_ENTITY, and WANDB_PROJECT from the user's environment or prompt.

Scope and approach

Classify each request before acting:

• Brief — a focused read or compute that maps to one query and a short answer ("How many runs?", "Best loss?", "Show the config of abc123"). Solve it in one script; add a second only if the first surfaced a load-bearing lead. Default to brief when in doubt.
• Intense — open-ended investigation with iterative discovery: ambiguous data, unknown schema, cross-cutting joins, plots, multi-stage analysis ("What's wrong with my training runs?", "Compare these sweeps"). Several scripts are fine, but each must be load-bearing — plan the next call from the data you just got, not from a generic checklist.

A W&B project has two complementary surfaces — runs (experiment tracking, wandb.Api()) and Weave traces (observability, weave.init() → client.get_calls()). For a broad "what's going on in this project?" question, probe both in the first evidence pass (parallel scripts, or the combined "Summarize project" recipe below), then scope the answer to the surface(s) that actually hold data. If a surface is empty, don't mention it.

Fast recipes — use these first

These cover the most common tasks. Each is a single script. Copy, fill in placeholders, run.

Fast product/API answers

For small W&B product or API questions, answer directly from this section. Do not run tools, inspect docs, or query the user's project unless they explicitly ask for live data. Keep the answer short: direct answer, exact UI/API path, minimal code if useful. If the recommendation depends on missing context, include targeted diagnostic questions in the same response instead of blocking.

For workspace migration or project-structure guidance, ask the diagnostic questions before prescribing a structure or script. Use the phrase "Before I prescribe a structure/script, I need to know:" and include the questions that materially change the answer; then give only tentative guidance.

Product facts to answer from memory

Trace-count semantics

Use these rules before every Weave count query:

• "total traces" or "total calls" means all calls. Use calls_query_stats with no trace_roots_only filter. Do not deduplicate by trace_id unless the prompt asks for unique traces.
• "root traces", "root-level traces", or "traces with no parent" means root calls. Use filter={"trace_roots_only": True} only for those prompts.
• "successful/non-error traces" means total calls minus calls with status error / descendant_error / non-null exception; report that as the primary count. summary.weave.status == "success" is a useful supporting breakdown, but it excludes running calls, which are still non-error. Do not count only root traces unless the user says root/root-level.
• "error/exception traces" means calls with status error OR descendant_error OR a non-null exception. For root-level error counts, add trace_roots_only=True to that same error query.
• Evaluation.evaluate counts are op counts. Use an op_names filter for weave:///<entity>/<project>/op/Evaluation.evaluate:*. Add trace_roots_only only if the user explicitly asks for root eval traces.
• For exact count tasks, run one script that prints the query and the number; do not run sample/exploratory scripts after the count is already known.

Eval-analysis rules

• Filter Evaluation.evaluate calls with op_names=[f"weave:///{entity}/{project}/op/Evaluation.evaluate:*"].
• Fetch only needed columns (id, display_name, started_at, ended_at, summary, inputs, output) and avoid broad object dumps.
• Eval token usage is in summary.usage; sum input_tokens, output_tokens, and total_tokens across model keys.
• Eval success/error counts are in summary.status_counts, not summary.weave.status_counts. Normalize enum and string keys before reading success, error, and descendant_error.
• For success-rate tasks, do not lead with a long 43-row markdown table. First answer with totals, both fractions, and a compact Error evaluations (N): TSV/code block containing every errored eval id, date, success_count, error_count, and status. If full per-eval rows are requested, use short IDs/dates/counts after the error list; avoid repeating long duplicate display names where they cause truncation. If some evals are still running, report both denominators: success-status evals over completed evals and no-error evals over all evals.
• Child dataset rows are Evaluation.predict_and_score:* calls with parent_ids=[eval_call.id].
• Dataset refs live on inputs["self"].dataset inside the Evaluation object. Count distinct dataset object refs from the user's project data; repeated evals can reuse the same dataset ref.
• For scorer inventories, eval summaries, and scorer evolution, include both wrapper scorer ops whose short names end in _scorer and class scorer ops ending in .score. Never filter only for the substring scorer; versioned class scorers like MyClassifier.score do not contain it.
• For large scorer inventories, include a compact full TSV/code block (scorer\tcount) for every scorer and then summarize family groupings. Do not use long prose tables that may truncate before all counts appear.

Count runs (exact, fast)

import wandb, os
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"
total = len(api.runs(path, per_page=1, include_sweeps=False, lazy=True))
finished = len(api.runs(path, filters={"state": "finished"}, per_page=1, include_sweeps=False, lazy=True))
crashed = len(api.runs(path, filters={"state": "crashed"}, per_page=1, include_sweeps=False, lazy=True))
running = len(api.runs(path, filters={"state": "running"}, per_page=1, include_sweeps=False, lazy=True))
print(f"Total: {total}  |  Finished: {finished}  |  Crashed: {crashed}  |  Running: {running}")

Run-count rules:

• Use one script for exact counts. If it prints the requested count, answer from that stdout; do not rerun just to add labels or nicer formatting.
• Use include_sweeps=False for normal run-table counts unless the prompt asks for sweep runs. For sweep counts, query sweeps explicitly.
• For status breakdowns, scan once and report all states you see (finished, failed, crashed, killed, etc.). When crashed/killed runs exist, report unsuccessful terminal rate (failed + crashed + killed) / total as the primary failure rate and include failed-only rate as a supporting number.
• For tags, count runs with at least one tag and also list distinct tag names and the runs attached to each tag.
• For run groups, report named groups from groupedRuns(groupKeys: ["group"]) and compute ungrouped runs as total_runs - sum(named_group_counts).
• For sweep-run tasks, list each sweep's run count and explicitly report the total runs across all sweeps.

Count/list sweeps

Do not inspect the W&B SDK source for routine sweep questions. Use the public project API directly:

import os, wandb

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
api = wandb.Api(timeout=120)

sweeps = list(api.project(project, entity=entity).sweeps(per_page=50))
rows = []
for sweep in sweeps:
    config = sweep.config or {}
    metric = config.get("metric") or {}
    rows.append({
        "id": sweep.id,
        "state": sweep.state,
        "method": config.get("method"),
        "metric": metric.get("name"),
        "goal": metric.get("goal"),
        "run_count": len(sweep.runs),
    })

print(f"sweep_count={len(rows)}")
print(f"total_sweep_runs={sum(r['run_count'] for r in rows)}")
for r in rows:
    print(r)

Finished runs with trigger/user

For prompts asking who triggered each run, fetch the filtered runs once and read run.user.username / run.user.name; do not search reference files.

import os, wandb

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
path = f"{entity}/{project}"
api = wandb.Api(timeout=120)

runs = api.runs(
    path,
    filters={"state": "finished"},
    order="+created_at",
    per_page=100,
    include_sweeps=False,
)
rows = []
for run in runs:
    user = getattr(run, "user", None)
    rows.append({
        "created_at": run.created_at,
        "name": run.display_name or run.name,
        "id": run.id,
        "username": getattr(user, "username", None),
        "user_name": getattr(user, "name", None),
    })

print(f"finished_count={len(rows)}")
for r in rows:
    print(r)

Count traces (fast, server-side)

import weave, os, logging
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq
from weave.trace_server.interface.query import Query

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
client = weave.init(f"{entity}/{project}")
pid = f"{entity}/{project}"

# Total calls/traces
stats = client.server.calls_query_stats(CallsQueryStatsReq(project_id=pid))
print(f"Total calls: {stats.count}")

# Root traces only
root_stats = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, filter={"trace_roots_only": True}
))
print(f"Root traces: {root_stats.count}")

# Count by op name
for op in ["Evaluation.evaluate", "my_op.turn"]:
    op_ref = f"weave:///{entity}/{project}/op/{op}:*"
    s = client.server.calls_query_stats(CallsQueryStatsReq(
        project_id=pid,
        filter={"op_names": [op_ref]},
    ))
    print(f"  {op}: {s.count}")

# Count calls whose op_name contains a substring, e.g. scorer calls.
score_query = Query(**{"$expr": {"$contains": {
    "input": {"$getField": "op_name"},
    "substr": {"$literal": ".score"},
    "case_insensitive": True,
}}})
score_stats = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, query=score_query
))
print(f"Scorer calls (.score): {score_stats.count}")

# Count a named op substring such as create_embeddings.
embedding_query = Query(**{"$expr": {"$contains": {
    "input": {"$getField": "op_name"},
    "substr": {"$literal": "create_embeddings"},
    "case_insensitive": True,
}}})
embedding_stats = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, query=embedding_query
))
print(f"create_embeddings calls: {embedding_stats.count}")

# Error/exception calls. Include descendant_error when the prompt says
# "error status or exception"; those are traces whose children failed.
error_query = Query(**{"$expr": {"$or": [
    {"$eq": [{"$getField": "summary.weave.status"}, {"$literal": "error"}]},
    {"$eq": [
        {"$getField": "summary.weave.status"},
        {"$literal": "descendant_error"},
    ]},
    {"$not": [{"$eq": [{"$getField": "exception"}, {"$literal": None}]}]},
]}})
error_stats = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, query=error_query
))
root_error_stats = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, filter={"trace_roots_only": True}, query=error_query
))
print(f"Error/exception calls: {error_stats.count}")
print(f"Root error/exception calls: {root_error_stats.count}")
print(f"Non-error calls: {stats.count - error_stats.count}")

Count create_embeddings calls and input sizes

import os, statistics, weave, logging, sys
from collections import Counter
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq
from weave.trace_server.interface.query import Query
sys.path.insert(0, "skills/wandb-primary/scripts")
from weave_helpers import unwrap

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)

query = Query(**{"$expr": {"$contains": {
    "input": {"$getField": "op_name"},
    "substr": {"$literal": "create_embeddings"},
    "case_insensitive": True,
}}})
total = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, query=query
)).count

sizes = []
for call in client.get_calls(query=query, limit=total, columns=["inputs"]):
    inputs = unwrap(call.inputs)
    texts = inputs.get("texts") or inputs.get("input") or []
    if isinstance(texts, str):
        sizes.append(1)
    else:
        sizes.append(len(texts))

dist = Counter(sizes)
print(f"create_embeddings calls: {total}")
print(f"typical texts per call: {dist.most_common(1)[0][0] if dist else 0}")
print(f"distribution: {dict(sorted(dist.items()))}")
print(f"mean texts per call: {statistics.mean(sizes) if sizes else 0:.4f}")

Count feedback records

import os, weave, logging
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import FeedbackQueryReq

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)

limit = 1000
offset = 0
total = 0
while True:
    res = client.server.feedback_query(FeedbackQueryReq(
        project_id=pid,
        fields=["id"],
        limit=limit,
        offset=offset,
    ))
    rows = (
        getattr(res, "result", None)
        or getattr(res, "feedback", None)
        or getattr(res, "rows", None)
        or []
    )
    n = len(rows)
    total += n
    if n < limit:
        break
    offset += limit

print(f"Feedback records: {total}")

List root op names with counts

import os, weave, logging
from collections import Counter
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)
root_filter = {"trace_roots_only": True}

root_count = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, filter=root_filter
)).count

def short_op(op_name: str) -> str:
    tail = op_name.split("/op/")[-1]
    return tail.rsplit(":", 1)[0]

counts = Counter()
for call in client.get_calls(
    filter=root_filter,
    limit=root_count,
    columns=["op_name"],
):
    counts[short_op(call.op_name)] += 1

for name, count in counts.most_common():
    print(f"{name}\t{count}")

List all op names with counts

import os, weave, logging
from collections import Counter
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)
total = client.server.calls_query_stats(CallsQueryStatsReq(project_id=pid)).count

def short_op(op_name: str) -> str:
    return op_name.split("/op/")[-1].rsplit(":", 1)[0]

counts = Counter()
for call in client.get_calls(limit=total, columns=["op_name"]):
    counts[short_op(call.op_name)] += 1

print(f"Unique ops: {len(counts)}")
for name, count in counts.most_common():
    print(f"{name}\t{count}")

Count long-duration traces

Do not try to do datetime arithmetic inside a Weave Query; stream timestamp columns and count locally.

import os, weave, logging
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)
total = client.server.calls_query_stats(CallsQueryStatsReq(project_id=pid)).count

threshold_s = 60
long_count = 0
scanned = 0
for call in client.get_calls(
    limit=total,
    columns=["started_at", "ended_at"],
):
    scanned += 1
    if call.started_at and call.ended_at:
        duration_s = (call.ended_at - call.started_at).total_seconds()
        if duration_s > threshold_s:
            long_count += 1

print(f"Scanned calls: {scanned}")
print(f"Duration > {threshold_s}s: {long_count}")

Find model names in traces

import os, weave, logging
from collections import Counter
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq
import sys
sys.path.insert(0, "skills/wandb-primary/scripts")
from weave_helpers import unwrap

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)
total = client.server.calls_query_stats(CallsQueryStatsReq(project_id=pid)).count

def collect_models(obj, out):
    obj = unwrap(obj)
    if isinstance(obj, dict):
        for k, v in obj.items():
            if k == "model" and isinstance(v, str):
                out.append(v)
            collect_models(v, out)
    elif isinstance(obj, list):
        for item in obj:
            collect_models(item, out)

models = Counter()
for call in client.get_calls(
    limit=total,
    columns=["inputs", "output", "summary"],
):
    found = []
    collect_models(call.inputs, found)
    collect_models(call.output, found)
    usage = unwrap(call.summary).get("usage", {}) if call.summary else {}
    for model_name in usage:
        if isinstance(model_name, str):
            found.append(model_name)
    for model_name in set(found):
        models[model_name] += 1

for name, count in models.most_common():
    print(f"{name}\t{count}")

Analyze embedding dimensions and model

import os, weave, logging
from collections import Counter
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.interface.query import Query
import sys
sys.path.insert(0, "skills/wandb-primary/scripts")
from weave_helpers import unwrap

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
client = weave.init(f"{entity}/{project}")

embedding_query = Query(**{"$expr": {"$contains": {
    "input": {"$getField": "op_name"},
    "substr": {"$literal": "create_embeddings"},
    "case_insensitive": True,
}}})

dims = Counter()
models = Counter()
no_output = 0
for call in client.get_calls(
    query=embedding_query,
    limit=100000,
    columns=["inputs", "output"],
):
    inputs = unwrap(call.inputs) or {}
    model = inputs.get("model") if isinstance(inputs, dict) else None
    models[model or "<missing>"] += 1

    output = unwrap(call.output)
    found = False
    if isinstance(output, list):
        for item in output:
            if isinstance(item, list) and item and isinstance(item[0], (int, float)):
                dims[len(item)] += 1
                found = True
    if not found:
        no_output += 1

print("embedding_models")
for name, count in models.most_common():
    print(f"{name}\t{count}")
print("embedding_dimensions")
for dim, count in dims.most_common():
    print(f"{dim}\t{count}")
print(f"no_embedding_output\t{no_output}")

List evaluation scorers

For scorer inventories, include wrapper scorer ops like faithfulness_scorer and class .score ops like HallucinationFreeScorer.score when present.

import os, weave, logging
from collections import Counter
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
pid = f"{entity}/{project}"
client = weave.init(pid)
total = client.server.calls_query_stats(CallsQueryStatsReq(project_id=pid)).count

def short_op(op_name: str) -> str:
    return op_name.split("/op/")[-1].rsplit(":", 1)[0]

scorers = Counter()
for call in client.get_calls(limit=total, columns=["op_name"]):
    name = short_op(call.op_name)
    if name.endswith("_scorer") or name.endswith(".score"):
        scorers[name] += 1

for name, count in scorers.most_common():
    print(f"{name}\t{count}")

Summarize project (runs + traces in one script)

import wandb, weave, os, logging
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace_server.trace_server_interface import CallsQueryStatsReq

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
path = f"{entity}/{project}"

# --- Runs ---
api = wandb.Api(timeout=120)
total_runs = len(api.runs(path, per_page=1, include_sweeps=False, lazy=True))
finished = len(api.runs(path, filters={"state": "finished"}, per_page=1, include_sweeps=False, lazy=True))
recent = api.runs(path, order="-created_at", per_page=5)[:5]

print(f"=== Runs ({total_runs} total, {finished} finished) ===")
for r in recent:
    print(f"  {r.name} [{r.state}] {r.created_at[:10]}")

# --- Weave Traces ---
client = weave.init(path)
pid = f"{entity}/{project}"
root_stats = client.server.calls_query_stats(CallsQueryStatsReq(
    project_id=pid, filter={"trace_roots_only": True}
))
print(f"\n=== Weave Traces ({root_stats.count} root traces) ===")

recent_calls = list(client.get_calls(
    sort_by=[{"field": "started_at", "direction": "desc"}],
    limit=5,
    columns=["op_name", "started_at", "display_name"],
))
for c in recent_calls:
    name = c.display_name or c.op_name.split("/")[-1].split(":")[0]
    started = c.started_at.strftime("%Y-%m-%d %H:%M") if c.started_at else "?"
    print(f"  {name} @ {started}")

Inspect a single run

import wandb, os
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"

run = api.run(f"{path}/RUN_ID")
print(f"Name: {run.name}")
print(f"State: {run.state}")
print(f"Created: {run.created_at}")
print(f"Tags: {run.tags}")
print(f"Last step: {run.lastHistoryStep}")

# Key metrics (replace with actual keys from probe or user request)
for k in ["loss", "val_loss", "accuracy"]:
    v = run.summary_metrics.get(k)
    if v is not None:
        print(f"  {k}: {v}")

Inventory artifacts (types → collections → versions)

The run table is not the whole project — artifacts (datasets, model checkpoints, tables) are separate. probe_project() surfaces artifact names; this enumerates them directly.

import wandb, os
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"

for atype in api.artifact_types(project=path):
    collections = list(api.artifact_collections(path, atype.name, per_page=1000))
    print(f"{atype.name}: {len(collections)} collections")
    for col in collections[:10]:
        try:
            n_versions = len(col.artifacts(per_page=50))
        except Exception:
            n_versions = "?"
        print(f"  {col.name}  ({n_versions} versions)")

Summarize an artifact's files (metadata + manifest + bounded read)

Inspect what an artifact contains — don't infer from its name. Read the manifest first; download only the small structured files you actually need.

import wandb, os
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"

art = api.artifact(f"{path}/ARTIFACT_NAME:latest")  # or :v3
print(f"name={art.name} type={art.type} size_bytes={art.size} aliases={art.aliases}")
md = art.metadata or {}
print(f"metadata_keys={list(md)[:20]}")

entries = sorted(art.manifest.entries.values(), key=lambda e: e.path)
print(f"file_count={len(entries)}")
for e in entries[:25]:
    print(f"  {e.path}  ({e.size} bytes)")

# Read ONE small structured file without downloading the whole artifact:
# p = art.get_entry("metrics.jsonl").download()  # local path to just that file
# import pandas as pd; df = pd.read_json(p, lines=True); print(df.describe())

Artifact rules:

• For "what's in this artifact?" read the manifest and a bounded sample of rows; do not download multi-GB artifacts to answer a structural question.
• Use run.logged_artifacts() to find a run's outputs (e.g. checkpoint locations) and run.used_artifacts() for its inputs.

System metrics (GPU / CPU / memory) — MUST use stream='system'

GPU, CPU, memory, network, and disk metrics live in a separate system stream. run.history() without stream='system' returns training metrics only — all system.gpu.*, system.cpu.*, system.memory.* keys will be absent. Finding no system keys in the default stream is NOT evidence they don't exist.

BEFORE concluding GPU or system metrics are unavailable, you MUST call run.history(stream='system').

import wandb, os, pandas as pd
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"

runs = api.runs(path, filters={"state": "finished"}, per_page=100)
rows = []
for run in runs:
    sys_df = run.history(stream="system", samples=500)
    if sys_df.empty or "system.gpu.0.gpu" not in sys_df.columns:
        rows.append({"run": run.name, "gpu_mean": None, "gpu_min": None, "gpu_max": None})
        continue
    gpu = sys_df["system.gpu.0.gpu"].dropna()
    rows.append({
        "run": run.name,
        "gpu_mean": round(gpu.mean(), 1),
        "gpu_min": round(gpu.min(), 1),
        "gpu_max": round(gpu.max(), 1),
        "low_util_pct": round(100 * (gpu < 30).sum() / len(gpu), 1) if len(gpu) else None,
    })

df = pd.DataFrame(rows)
print(df.to_string(index=False))

Run-lookup rules:

• For user-facing run names, prefer run.display_name or run.name; include run.id separately if useful. Do not report only the run ID as the name.
• For "best", "highest", "lowest", "latest", and "longest" tasks, use one script that prints name, id, metric value, state, group, job_type, and tags for the winner. Use that context in the final answer.
• For baseline-vs-hyperopt questions, group is None and empty job_type/tags usually indicate an ungrouped baseline; hyperopt trials usually have a named group and/or job_type="hyperopt". If the winning run is ungrouped while the runner-up runs are grouped hyperopt trials, state that explicitly.
• For final metric questions, check the summary metric first; use scan_history(keys=[...]) only if the summary is absent or the task explicitly asks for history.
• For config/model-variant questions, try api.runs(..., lazy=False) and GraphQL config reads. If configs are empty, say that and use run names, tags, groups, job_type, or files as the source; do not invent config values.
• For YOLOv5 weight inventories, normalize raw filenames such as yolov5s.pt to canonical variant names like yolov5s in the final count table; include a raw/source column when useful.

Run-analysis / project-summary rules:

• For project summaries, run one script that prints observed run counts, config keys/value frequencies, metric-key families, artifact types, and sweep status. In the final answer, only cite exact run IDs, metric values, or config values that were printed by the script; otherwise keep the summary at the observed high-level pattern.
• For project-specific summaries, do not rely on memorized project facts. Run the relevant W&B/Weave queries, print compact evidence, and ground the final answer only in the observed data.
• For outlier analysis, compute the requested metric/history statistics from the user's runs and make the top observed outlier the headline only when the evidence supports it.

OpenAI + Weave tracing setup:

• For OpenAI tracing setup questions, explicitly mention OpenAI auto-tracing: after weave.init(...), supported OpenAI client calls are automatically traced by Weave, or the user can use weave.integrations.openai.OpenAI. State that prompts, responses, token usage, latency, and errors are logged; use @weave.op() around app functions to add the app-level call tree.

W&B Sweep setup:

• For sweep setup questions, always show the concrete lifecycle in code: define a sweep config with method, metric, and parameters; create it via sweep_id = wandb.sweep(sweep_config, project=...); run agents via wandb.agent(sweep_id, function=train, count=...); and log metrics inside the training function with wandb.init(config=...) and wandb.log(...).
• Discuss grid, random, and bayesian search explicitly: grid for tiny discrete spaces, random for broad/cheap exploration and log-scale learning rates, bayesian for expensive refinement after the metric is stable. Mention parallel coordinates, parameter importance, sorted run tables, and rerunning the top configs/seeds before selecting a winner.

Diagnose training history (curves, spikes, NaNs, stability)

For "is training stable?" / "which runs diverged?" / "any loss spikes?", scan a metric's history across runs and compute stability stats locally. Always pass keys=[...]; for runs with 10K+ steps use beta_scan_history instead of history.

import wandb, os, numpy as np, pandas as pd
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"
metric = "train/loss"  # discover the real key first (probe_project / inspect a run)

runs = api.runs(path, filters={"state": "finished"}, per_page=100)[:40]
rows = []
for run in runs:
    df = run.history(samples=300, keys=[metric])  # never omit keys on large runs
    series = df[metric].dropna() if metric in getattr(df, "columns", []) else pd.Series(dtype=float)
    arr = series.to_numpy(dtype=float)
    finite = arr[np.isfinite(arr)]
    diffs = np.abs(np.diff(finite)) if finite.size > 2 else np.array([])
    spike_threshold = 5 * (np.median(diffs) or 1.0)
    rows.append({
        "run": run.display_name or run.name,
        "id": run.id,
        "points": int(arr.size),
        "nan_or_inf": int((~np.isfinite(arr)).sum()),
        "min": round(float(finite.min()), 5) if finite.size else None,
        "final": round(float(finite[-1]), 5) if finite.size else None,
        "spikes": int((diffs > spike_threshold).sum()),
    })

out = pd.DataFrame(rows).sort_values("min", na_position="last")
print(out.to_string(index=False))

min is the best value over history (not the endpoint); a large final - min gap, nonzero nan_or_inf, or many spikes flags an unstable or diverged run. For GPU under-utilization use the system-stream recipe above.

Compare two runs

import wandb, os, sys
sys.path.insert(0, "skills/wandb-primary/scripts")
from wandb_helpers import get_api, compare_configs

api = get_api()
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"

run_a = api.run(f"{path}/RUN_A_ID")
run_b = api.run(f"{path}/RUN_B_ID")

# Config diff
diffs = compare_configs(run_a, run_b)
if diffs:
    print("Config differences:")
    for d in diffs:
        print(f"  {d['key']}: {d[run_a.name]} -> {d[run_b.name]}")
else:
    print("Configs are identical")

# Metric comparison
print("\nMetrics:")
for k in ["loss", "val_loss", "accuracy"]:
    a = run_a.summary_metrics.get(k, "N/A")
    b = run_b.summary_metrics.get(k, "N/A")
    print(f"  {k}: {a} vs {b}")

Compare cohorts / variants (group by a config or run axis)

For "which variant/optimizer/group is best?", bucket runs by an axis (a config key, run.group, or run.job_type) and compare a metric across buckets. Report the full ladder, not just best and worst.

import wandb, os, numpy as np, pandas as pd
from collections import defaultdict
api = wandb.Api(timeout=120)
path = f"{os.environ['WANDB_ENTITY']}/{os.environ['WANDB_PROJECT']}"
metric = "accuracy"   # discover the real key first
axis = "optimizer"    # a config key; or use run.group / run.job_type

runs = api.runs(path, filters={"state": "finished"}, per_page=200)[:200]
buckets = defaultdict(list)
for run in runs:
    key = run.config.get(axis, "<missing>")  # or: run.group / run.job_type
    value = run.summary_metrics.get(metric)
    if value is not None:
        buckets[str(key)].append(float(value))

rows = [
    {axis: key, "n": len(vals), "mean": round(np.mean(vals), 4),
     "min": round(np.min(vals), 4), "max": round(np.max(vals), 4)}
    for key, vals in buckets.items()
]
out = pd.DataFrame(rows).sort_values("mean", ascending=False)
print(out.to_string(index=False))

If configs come back empty, the runs were fetched lazily — re-fetch with api.runs(..., per_page=200) and access config per run, or fall back to run.group/run.job_type/tags as the axis. Don't invent axis values.

Summarize latest eval

import weave, os, sys, logging
logging.getLogger("weave").setLevel(logging.ERROR)
from weave.trace.weave_client import CallsFilter
sys.path.insert(0, "skills/wandb-primary/scripts")
from weave_helpers import unwrap, eval_results_to_dicts, results_summary

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
client = weave.init(f"{entity}/{project}")

# Get latest eval
op_ref = f"weave:///{entity}/{project}/op/Evaluation.evaluate:*"
evals = list(client.get_calls(
    filter=CallsFilter(op_names=[op_ref]),
    sort_by=[{"field": "started_at", "direction": "desc"}],
    limit=1,
))

if not evals:
    print("No evaluations found")
else:
    ec = evals[0]
    print(f"Eval: {ec.display_name or 'unnamed'} @ {ec.started_at}")

    # Get predict_and_score children
    pas_ref = f"weave:///{entity}/{project}/op/Evaluation.predict_and_score:*"
    pas = list(client.get_calls(
        filter=CallsFilter(op_names=[pas_ref], parent_ids=[ec.id])
    ))
    results = eval_results_to_dicts(pas, agent_name=ec.display_name or "agent")
    print(results_summary(results))

Inspect recent traces

import weave, os, logging, sys
logging.getLogger("weave").setLevel(logging.ERROR)
sys.path.insert(0, "skills/wandb-primary/scripts")
from weave_helpers import unwrap, get_token_usage

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]
client = weave.init(f"{entity}/{project}")

calls = list(client.get_calls(
    sort_by=[{"field": "started_at", "direction": "desc"}],
    limit=10,
))

for c in calls:
    name = c.display_name or c.op_name.split("/")[-1].split(":")[0]
    started = c.started_at.strftime("%Y-%m-%d %H:%M") if c.started_at else "?"
    duration = ""
    if c.started_at and c.ended_at:
        duration = f" ({(c.ended_at - c.started_at).total_seconds():.1f}s)"
    status = c.summary.get("weave", {}).get("status", "?") if c.summary else "?"
    tokens = get_token_usage(c)
    tok_str = f" [{tokens['total_tokens']} tok]" if tokens['total_tokens'] else ""
    print(f"  {name} [{status}] {started}{duration}{tok_str}")

Create a W&B Report

Use wandb-workspaces for programmatic report definitions. For runset filters, panels, loading, and sharing, see references/REPORTS.md.

import os

import wandb_workspaces.reports.v2 as wr

entity = os.environ["WANDB_ENTITY"]
project = os.environ["WANDB_PROJECT"]

runset = wr.Runset(entity=entity, project=project, name="All runs")
plots = wr.PanelGrid(
    runsets=[runset],
    panels=[
        wr.LinePlot(title="Loss", x="_step", y=["LOSS_KEY"]),
        wr.BarPlot(title="Accuracy", metrics=["ACC_KEY"], orientation="v"),
    ],
)

report = wr.Report(
    entity=entity,
    project=project,
    title="Project Analysis",
    description="Auto-generated summary",
    width="fixed",
    blocks=[
        wr.H1("Project Analysis"),
        wr.P("Auto-generated summary from W&B API."),
        plots,
    ],
)
report.save(draft=True)
print(f"Report saved: {report.url}")

A clean report.save() return is not proof the report landed — saves can fail silently. For anything beyond a throwaway draft, save through report_helpers so you get a verified read-back instead of assuming success:

import sys
sys.path.insert(0, "skills/wandb-primary/scripts")
from report_helpers import save_report_verified

result = save_report_verified(report)  # draft=True by default
print(result["answer"])                # answer=... verified=True/False url=...

Launch

Use skills/wandb-primary/scripts/launch_helpers.py. Do not train locally to test GPU work, and do not fake Launch with a local wandb.init().

Every Launch entrypoint you create must call wandb.init(...),

…(truncated)