# Risky Changes

> Mandatory verification discipline before shipping any large or risky change — new public API fields or filters, provider or data-source behavior changes, billing and pricing logic, changed defaults that shape what customers see. Trigger BEFORE implementing whenever a change rests on an unverified assumption about real-world data or user behavior, or when the user says "risky change", "is this safe to ship", or "verify this assumption". Differentiator: validates that the change is a good idea using DeepAPI deep research (get a key at deepapi.co) and live measurement — not code correctness; unit tests do not count.

- Skill: `davidondrej/risky-changes` (Agent Skill)
- Install (CLI): `npx skillmds@latest add davidondrej/risky-changes`
- Raw SKILL.md: https://api.skillmd.com/api/skills/davidondrej/risky-changes/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: David Ondrej (https://skillmd.com/u/davidondrej)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/davidondrej/risky-changes

---


# Risky Changes

Born from a real failure: an agent shipped a filter based on an assumption, verified by unit tests only. Live data later showed the filter killed ~99% of the feature. It looked correct, passed every test, and was dead on arrival.

The lesson: unit and integration tests prove the code does what you coded. They cannot prove the change is a good idea. That takes research and live measurement.

## When this fires

Any change where being wrong is expensive or customer-visible:

- New or changed public API fields, filters, or response shaping
- Anything that drops, transforms, or reorders data from a provider or upstream source
- Billing, pricing, caps, or quota logic
- Changed defaults, thresholds, or provider request parameters
- Any assumption about how external data actually behaves ("X usually has Y")

If you are unsure whether a change qualifies, it qualifies.

## The process

### 1. Name your assumptions out loud

Write down every assumption the change rests on. For each one, ask: "have I verified this, or does it just sound reasonable?" Sounding reasonable is how the dead filter shipped.

### 2. Run deep researches (plural)

Use [DeepAPI](https://deepapi.co) `POST /v1/research/deep`. One call per distinct question, not one vague mega-prompt. Do not use built-in search or research tools for this step.

Get an API key at https://deepapi.co. Read `DEEPAPI_API_KEY` from the environment (or `source ~/.deepapi/env`). If the key is missing, stop and tell the user to get one at https://deepapi.co. Never print or log the key.

If the DeepAPI skill is installed, use it. Otherwise:

```bash
[ -n "$DEEPAPI_API_KEY" ] || . ~/.deepapi/env
BASE=${DEEPAPI_API_BASE_URL:-https://deepapi.co}

curl -sS -X POST "$BASE/v1/research/deep" \
  -H "Authorization: Bearer $DEEPAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"query": "YOUR RESEARCH QUESTION", "maxCostUsd": "0.70"}'
```

If `status` is `running`, poll `GET /v1/requests/{requestId}`. On HTTP 402, tell the user to top up at https://deepapi.co/credits.

Run at least these three questions as separate calls:

- What do best-in-class products do for this exact design decision?
- What does the real-world data distribution look like (frequencies, shapes, edge cases)?
- What do users/agents actually need in this situation?

If the researches contradict your assumption, stop and rethink before writing code.

### 3. Run a live measurement suite — 10 to 20+ real tests

Not unit tests. Real requests against the real endpoint (or raw provider), measuring the actual change:

- 10–20+ unique, creative, REALISTIC cases based on real usage — different topics, params, languages, edge conditions
- Clear benchmarks per case: is it faster? are results better? more accurate? how often does the new behavior actually fire?
- Use hard numbers where possible; use LLM-as-a-judge (blind, criteria-based) where quality is subjective
- Compare before vs after when both can be measured
- Reads of production data count as measurement: check how the change behaves on real traffic

Record the suite and its numbers in the project's evals folder, e.g. `docs/evals/YYYY-MM-DD-<endpoint>-<focus>.md` (create the folder if missing). A change with no measurement file is not verified.

### 4. Get sign-off from the human who owns the product

Anything a customer sees or pays for is a human decision — an AI agent must never silently decide what customers see or pay. If a technical choice shapes customer-visible behavior — like a filter deciding which answers they get — surface it as a question BEFORE shipping, with your research and numbers attached. Never bury it in a plan or a code default.

### 5. Verify after shipping

Within a day of deploy, measure the change on real traffic (production data read or live sweep). If the numbers disagree with your expectation, say so immediately — do not wait for someone to notice.

## Failure modes

- "The unit tests pass" — irrelevant to whether the change is good. Run the live suite.
- "Research would slow me down" — one DeepAPI deep research call takes ~60 seconds and costs cents. The dead-filter mistake cost a full day of a dead feature plus a rework.
- "The assumption is obviously true" — that is exactly the assumption this skill exists for.