Benchmark Category Development

Benchmark Category Development

Purpose

Use this repo-specific workflow to develop benchmark task categories such as DB access-control, integrity, query, vector, storage, or auth. The goal is a clean, central, provider-neutral task set with trustworthy verifier results and reproducible workflow comparisons.

Core Rules

• Keep tasks central and provider-neutral under tasks/<module>/<category>/<task_id>/.
• Do not preserve legacy paths, compatibility fallbacks, pilot concepts, or historical overlays.
• Prompts describe the product goal and expected backend contract only. Workflow/tool restrictions belong in workflow profiles, not task prompts.
• Verifiers test observable behavior, security properties, and production readiness. Do not require a specific implementation unless the task is explicitly about that implementation contract.
• Use attack scripts for negative/security probes when the task can have false-positive pass cases.
• Treat failures as data only after checking whether they are real agent failures, verifier false negatives, verifier false positives, harness bugs, timeout hangs, or isolation-rule issues.
• Sandbox is the default. Do not depend on the user's local dev containers or external provider projects.
• Avoid adding benchmark-specific instructions to product skills. Task prompts may state fair domain constraints, such as DB work staying in public schema when the benchmark target only supports that schema.

Category Design

Before writing tasks, define the category boundary in one sentence.

Use these DB category meanings unless the user changes them:

• access-control: row, relation, field, role, ownership, and sharing authorization.
• integrity: invariants, lifecycle rules, constraints, triggers, derived state, state transitions, and maintenance logic.
• query: reporting, aggregation, search, pagination, query-facing views/functions, and query performance as an optional metric.
• vector: embeddings, similarity retrieval, hybrid search, metadata filters, and ranking.

Build a candidate list, then reduce it. Keep only tasks that cover a distinct failure mode or product capability. Merge only when the combined task is still a natural real-world scenario, not two unrelated tasks glued together.

For each category, prefer fewer high-quality tasks over many repetitive ones. Difficulty labels should reflect expected agent complexity and runtime:

• simple: one core concept, quick implementation, useful as a baseline.
• medium: two or three interacting requirements, likely one non-obvious edge case.
• hard: realistic multi-step backend behavior, but still designed to finish within about 5 minutes.

Task Authoring Checklist

Each task directory should normally contain:

meta.json
prompt.md
prepare.ts
verify.ts
attack.ts
cleanup.ts
state.ts

Use run.ts only when a compact single-file task is clearly cleaner.

meta.json:

• suite is the module suite, such as db.
• capability equals the category directory name.
• difficulty is simple, medium, or hard.
• timeoutMs should usually be 300000 for agent-backed benchmark tasks.

prompt.md:

• Tell the agent to read task-state.json.
• Use exact names from state.
• Keep the task provider-neutral.
• Avoid direct HTTP/API/client test instructions; verifier owns user registration, login, uploads, reads, attacks, and production-readiness checks.
• Require implementation.json with useful completion notes.

prepare.ts:

• Create deterministic randomized names and all seed users/data needed by verifier.
• Keep provider differences out unless the provider architecture truly requires it.

verify.ts:

• Exercise the intended production path using realistic client/admin behavior.
• Assert positive behavior, negative behavior, and edge cases.
• Prefer outcome checks over introspecting exact SQL, policy names, or implementation shape.
• Include diagnostics that make failures understandable without reading every log line.

attack.ts:

• Try bypasses that a broken implementation might accidentally allow.
• For access control, include cross-user, cross-tenant, privilege escalation, protected-field mutation, and stale-access attempts when relevant.

cleanup.ts:

• Drop task-created objects and data only.
• Be safe to run after partial agent failure.

Calibration Workflow

1. Run npm run typecheck.
2. Run one task at k=1 on the most relevant InsForge workflow.
3. If it fails, inspect logs before changing anything.
4. Classify the failure:
   • real agent failure
   • verifier false negative
   • verifier false positive risk
   • harness/sandbox issue
   • timeout/hang before meaningful work
   • workflow isolation rule issue
5. Fix only task or harness issues. Do not tune prompts or skills just to make one task pass.
6. Re-run the affected task after each fix.
7. Once each task is credible, run the selected cross-provider workflow set.

Useful command shapes:

npm run typecheck
npm run benchmark:k -- --k 1 --provider insforge --workflow cli-skills --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider insforge --workflow mcp --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider supabase --workflow cli-skills --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider supabase --workflow mcp --module db --capability <category> --task <task_id> --run-id <run_id>
npm run benchmark:k -- --k 1 --provider supabase --workflow mcp-skills --module db --capability <category> --task <task_id> --run-id <run_id>

Run workflows one at a time for large k experiments unless the user explicitly wants a matrix shortcut. This keeps memory use predictable and makes interrupted runs easier to resume.

Full K-Run Workflow

For a finalized category:

1. Run k=3 for each selected workflow, one workflow at a time.
2. After each workflow, scan failures immediately.
3. If a result is invalid because of harness/verifier/sandbox problems, fix the issue and rerun only the affected task/workflow/attempt set.
4. Keep the latest valid result set and remove stale invalid runs when the user asks for cleanup.
5. Generate or update a concise report with overall results, task-level reliability, time, token, cache-token rate when available, and failure notes.

For the current RLS comparison, the common selected workflows are:

• InsForge CLI+Skills
• InsForge MCP
• Supabase CLI+Skills
• Supabase MCP
• Supabase MCP+Skills

Do not assume this list applies to every category; confirm the intended comparison surface from the current request and repo state.

Report Shape

Write reports like a benchmark report, not a work log.

Include:

• scope and date
• workflow table with pass@k, pass^k, average time, P95 time, average tokens, and cache-token rate when available
• task-level table sorted by difficulty and task id
• concise failure-pattern analysis
• task summary table with goal and primary verifier expectation
• next steps grounded in observed failures

Avoid:

• run IDs unless the user asks for raw reproducibility details
• replacement-task/process history unless it affects interpretation
• compliance internals that require long explanation in public-facing reports
• personal correction notes or chat-style apologies