# Benchmark Category Development

> Use when designing, implementing, calibrating, running, or reporting task categories in the insforge-agent-benchmark repo.

- Skill: `insforge/benchmark-category-development` (Agent Skill)
- Install (CLI): `npx skillmds@latest add insforge/benchmark-category-development`
- Raw SKILL.md: https://api.skillmd.com/api/skills/insforge/benchmark-category-development/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: insforge (https://skillmd.com/u/insforge)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/insforge/benchmark-category-development

---


# Benchmark Category Development

## Purpose

Use this repo-specific workflow to develop benchmark task categories such as DB access-control, integrity, query, vector, storage, or auth. The goal is a clean, central, provider-neutral task set with trustworthy verifier results and reproducible workflow comparisons.

## Core Rules

- Keep tasks central and provider-neutral under `tasks/<module>/<category>/<task_id>/`.
- Do not preserve legacy paths, compatibility fallbacks, pilot concepts, or historical overlays.
- Prompts describe the product goal and expected backend contract only. Workflow/tool restrictions belong in workflow profiles, not task prompts.
- Verifiers test observable behavior, security properties, and production readiness. Do not require a specific implementation unless the task is explicitly about that implementation contract.
- Use attack scripts for negative/security probes when the task can have false-positive pass cases.
- Treat failures as data only after checking whether they are real agent failures, verifier false negatives, verifier false positives, harness bugs, timeout hangs, or isolation-rule issues.
- Sandbox is the default. Do not depend on the user's local dev containers or external provider projects.
- Avoid adding benchmark-specific instructions to product skills. Task prompts may state fair domain constraints, such as DB work staying in `public` schema when the benchmark target only supports that schema.

## Category Design

Before writing tasks, define the category boundary in one sentence.

Use these DB category meanings unless the user changes them:

- `access-control`: row, relation, field, role, ownership, and sharing authorization.
- `integrity`: invariants, lifecycle rules, constraints, triggers, derived state, state transitions, and maintenance logic.
- `query`: reporting, aggregation, search, pagination, query-facing views/functions, and query performance as an optional metric.
- `vector`: embeddings, similarity retrieval, hybrid search, metadata filters, and ranking.

Build a candidate list, then reduce it. Keep only tasks that cover a distinct failure mode or product capability. Merge only when the combined task is still a natural real-world scenario, not two unrelated tasks glued together.

For each category, prefer fewer high-quality tasks over many repetitive ones. Difficulty labels should reflect expected agent complexity and runtime:

- `simple`: one core concept, quick implementation, useful as a baseline.
- `medium`: two or three interacting requirements, likely one non-obvious edge case.
- `hard`: realistic multi-step backend behavior, but still designed to finish within about 5 minutes.

## Task Authoring Checklist

Each task directory should normally contain:

```text
meta.json
prompt.md
prepare.ts
verify.ts
attack.ts
cleanup.ts
state.ts
```

Use `run.ts` only when a compact single-file task is clearly cleaner.

`meta.json`:

- `suite` is the module suite, such as `db`.
- `capability` equals the category directory name.
- `difficulty` is `simple`, `medium`, or `hard`.
- `timeoutMs` should usually be `300000` for agent-backed benchmark tasks.

`prompt.md`:

- Tell the agent to read `task-state.json`.
- Use exact names from state.
- Keep the task provider-neutral.
- Avoid direct HTTP/API/client test instructions; verifier owns user registration, login, uploads, reads, attacks, and production-readiness checks.
- Require `implementation.json` with useful completion notes.

`prepare.ts`:

- Create deterministic randomized names and all seed users/data needed by verifier.
- Keep provider differences out unless the provider architecture truly requires it.

`verify.ts`:

- Exercise the intended production path using realistic client/admin behavior.
- Assert positive behavior, negative behavior, and edge cases.
- Prefer outcome checks over introspecting exact SQL, policy names, or implementation shape.
- Include diagnostics that make failures understandable without reading every log line.

`attack.ts`:

- Try bypasses that a broken implementation might accidentally allow.
- For access control, include cross-user, cross-tenant, privilege escalation, protected-field mutation, and stale-access attempts when relevant.

`cleanup.ts`:

- Drop task-created objects and data only.
- Be safe to run after partial agent failure.

## Calibration Workflow

1. Run `npm run typecheck`.
2. Run one task at `k=1` on the most relevant InsForge workflow.
3. If it fails, inspect logs before changing anything.
4. Classify the failure:
   - real agent failure
   - verifier false negative
   - verifier false positive risk
   - harness/sandbox issue
   - timeout/hang before meaningful work
   - workflow isolation rule issue
5. Fix only task or harness issues. Do not tune prompts or skills just to make one task pass.
6. Re-run the affected task after each fix.
7. Once each task is credible, run the selected cross-provider workflow set.

Useful command shapes:

```bash
npm run typecheck
npm run benchmark:k -- --k 1 --provider insforge --workflow cli-skills --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider insforge --workflow mcp --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider supabase --workflow cli-skills --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider supabase --workflow mcp --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider supabase --workflow mcp-skills --module db --capability <category> --task <task_id> --run-id <run_id>
```

Run workflows one at a time for large `k` experiments unless the user explicitly wants a matrix shortcut. This keeps memory use predictable and makes interrupted runs easier to resume.

## Full K-Run Workflow

For a finalized category:

1. Run `k=3` for each selected workflow, one workflow at a time.
2. After each workflow, scan failures immediately.
3. If a result is invalid because of harness/verifier/sandbox problems, fix the issue and rerun only the affected task/workflow/attempt set.
4. Keep the latest valid result set and remove stale invalid runs when the user asks for cleanup.
5. Generate or update a concise report with overall results, task-level reliability, time, token, cache-token rate when available, and failure notes.

For the current RLS comparison, the common selected workflows are:

- InsForge CLI+Skills
- InsForge MCP
- Supabase CLI+Skills
- Supabase MCP
- Supabase MCP+Skills

Do not assume this list applies to every category; confirm the intended comparison surface from the current request and repo state.

## Report Shape

Write reports like a benchmark report, not a work log.

Include:

- scope and date
- workflow table with pass@k, pass^k, average time, P95 time, average tokens, and cache-token rate when available
- task-level table sorted by difficulty and task id
- concise failure-pattern analysis
- task summary table with goal and primary verifier expectation
- next steps grounded in observed failures

Avoid:

- run IDs unless the user asks for raw reproducibility details
- replacement-task/process history unless it affects interpretation
- compliance internals that require long explanation in public-facing reports
- personal correction notes or chat-style apologies