# Designing Bench Task

> Use when designing a new bench_env task suite, adding several new tasks to an existing suite, or critiquing a task-set proposal for a mobile-gym App — before any `class FooTask(...)` is written under `bench_env/task/`.

- Skill: `purewhiter/designing-bench-task` (Agent Skill)
- Install (CLI): `npx skillmds@latest add purewhiter/designing-bench-task`
- Raw SKILL.md: https://api.skillmd.com/api/skills/purewhiter/designing-bench-task/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: purewhiter (https://skillmd.com/u/purewhiter)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/purewhiter/designing-bench-task

---


# Designing bench_env Tasks

## Overview

Rushing from "here's an App" to "here are 5 task classes" produces low-difficulty suites whose judge logic can't actually verify completion. Design must precede code.

**Authoritative reference:** `bench_env/docs/task/TASK_AUTHORING_GUIDE.md` (reading §1 + §2 once is required; this skill enforces its gates).

## The Gate: two artifacts before any Python

Produce both as plain text in the conversation **before** writing any task class. If you catch yourself opening `tasks.py` / `defs/<TaskName>.py`, stop and produce them.

### 1. Functional audit table (TASK_AUTHORING_GUIDE §1.1–1.2)

A table with one row per distinct feature area. Columns:

| Page/feature | Source file(s) | User-visible actions | Observable state path |

You must actually read: `manifest.ts`, `navigation.declaration.ts`, `data/defaults.json`, `state.ts`, `pages/*`, and the suite's `app.py` accessor if it exists. No skipping "because the app looks simple."

### 2. Data sufficiency check (TASK_AUTHORING_GUIDE §1.3)

For every function you plan to parameterize, confirm `defaults.json` / `state.ts` provides ≥3 varied entries. If it doesn't, either propose expanding defaults, or drop parameterization for that function.

## Per-task: 4 judge-predict questions

For each proposed task, answer in 1-2 lines each **before writing code** (this is the soundness/completeness audit later enforced by `TASK_AUTHORING_GUIDE §2.7` "Reliability requirements"):

1. Agent 完全做对时，最终 state / answer 长什么样？
2. Agent 最常见的 1-2 种错误是什么？会不会被误判通过？（soundness）
3. 有没有合理完成任务的替代路径？会不会被误判失败？（completeness）
4. 有无边界情况导致正确答案不唯一、或判定证据不足？

If any answer surfaces a flaw (common: initial state already equals criteria; ground truth not unique; answer requires subjective judgement), **iterate the design in text** — do not defer the fix to code review.

## Rationalization table — STOP and do the step

| Excuse | Reality |
|---|---|
| "App is tiny, audit is overkill" | Audit surfaces the data gap so you can close it before writing code. |
| "Judge predict is slow, I'll see issues when coding" | Design bugs (init=goal, non-unique ground truth) are 10× cheaper to fix in text. |
| "These tasks are obvious, pre-sim is busywork" | A task obvious enough to skip pre-sim is obvious enough to answer the 4 questions in 30 seconds. |
| "I'll produce both artifacts and code in one pass" | Then when code inherits a design flaw, you've wasted the coding pass. Gate is gate. |

## Checklist before producing any `class FooTask(...)`

- [ ] Functional audit table in conversation
- [ ] Data sufficiency assessed per parameterized function; gaps declared
- [ ] For each task, 4 pre-sim questions answered
- [ ] No task has initial state == goal state (the task must require Agent action to flip something)
- [ ] Cross-checked `TASK_AUTHORING_GUIDE.md` §2.7 "Reliability requirements" + §4.7 "Authoring check_goals"

## After design is approved

For actual code discipline: see the **writing-bench-task-judge** skill and `bench_env/docs/task/TASK_CODE_SPEC.md`. For tests: **testing-bench-task** skill + `bench_env/docs/task/TASK_TESTING_GUIDE.md`.