/ar:setup — Create New Experiment
Set up a new autoresearch experiment with all required configuration.
Usage
/ar:setup # Interactive mode
/ar:setup engineering api-speed src/api.py "pytest bench.py" p50_ms lower
/ar:setup --list # Show existing experiments
/ar:setup --list-evaluators # Show available evaluators
What It Does
If arguments provided
Pass them directly to the setup script:
python {skill_path}/scripts/setup_experiment.py \
--domain {domain} --name {name} \
--target {target} --eval "{eval_cmd}" \
--metric {metric} --direction {direction} \
[--evaluator {evaluator}] [--scope {scope}]
If no arguments (interactive mode)
Collect each parameter one at a time:
- Domain — Ask: "What domain? (engineering, marketing, content, prompts, custom)"
- Name — Ask: "Experiment name? (e.g., api-speed, blog-titles)"
- Target file — Ask: "Which file to optimize?" Verify it exists.
- Eval command — Ask: "How to measure it? (e.g., pytest bench.py, python evaluate.py)"
- Metric — Ask: "What metric does the eval output? (e.g., p50_ms, ctr_score)"
- Direction — Ask: "Is lower or higher better?"
- Evaluator (optional) — Show built-in evaluators. Ask: "Use a built-in evaluator, or your own?"
- Scope — Ask: "Store in project (.autoresearch/) or user (~/.autoresearch/)?"
Then run setup_experiment.py with the collected parameters.
Listing
# Show existing experiments
python {skill_path}/scripts/setup_experiment.py --list
# Show available evaluators
python {skill_path}/scripts/setup_experiment.py --list-evaluators
Built-in Evaluators
| Name |
Metric |
Use Case |
benchmark_speed |
p50_ms (lower) |
Function/API execution time |
benchmark_size |
size_bytes (lower) |
File, bundle, Docker image size |
test_pass_rate |
pass_rate (higher) |
Test suite pass percentage |
build_speed |
build_seconds (lower) |
Build/compile/Docker build time |
memory_usage |
peak_mb (lower) |
Peak memory during execution |
llm_judge_content |
ctr_score (higher) |
Headlines, titles, descriptions |
llm_judge_prompt |
quality_score (higher) |
System prompts, agent instructions |
llm_judge_copy |
engagement_score (higher) |
Social posts, ad copy, emails |
After Setup
Report to the user:
- Experiment path and branch name
- Whether the eval command worked and the baseline metric
- Suggest: "Run
/ar:run {domain}/{name} to start iterating, or /ar:loop {domain}/{name} for autonomous mode."
Creator: Engineering
License: MIT
Source Repo: neekware/dojo-skills
Source Bucket: engineering
Original Path: engineering/autoresearch-agent/skills/setup
1---2name: setup-23description: Setup4---5# /ar:setup — Create New Experiment67Set up a new autoresearch experiment with all required configuration.89## Usage1011```12/ar:setup # Interactive mode13/ar:setup engineering api-speed src/api.py "pytest bench.py" p50_ms lower14/ar:setup --list # Show existing experiments15/ar:setup --list-evaluators # Show available evaluators16```1718## What It Does1920### If arguments provided2122Pass them directly to the setup script:2324```bash25python {skill_path}/scripts/setup_experiment.py \26 --domain {domain} --name {name} \27 --target {target} --eval "{eval_cmd}" \28 --metric {metric} --direction {direction} \29 [--evaluator {evaluator}] [--scope {scope}]30```3132### If no arguments (interactive mode)3334Collect each parameter one at a time:35361. **Domain** — Ask: "What domain? (engineering, marketing, content, prompts, custom)"372. **Name** — Ask: "Experiment name? (e.g., api-speed, blog-titles)"383. **Target file** — Ask: "Which file to optimize?" Verify it exists.394. **Eval command** — Ask: "How to measure it? (e.g., pytest bench.py, python evaluate.py)"405. **Metric** — Ask: "What metric does the eval output? (e.g., p50_ms, ctr_score)"416. **Direction** — Ask: "Is lower or higher better?"427. **Evaluator** (optional) — Show built-in evaluators. Ask: "Use a built-in evaluator, or your own?"438. **Scope** — Ask: "Store in project (.autoresearch/) or user (~/.autoresearch/)?"4445Then run `setup_experiment.py` with the collected parameters.4647### Listing4849```bash50# Show existing experiments51python {skill_path}/scripts/setup_experiment.py --list5253# Show available evaluators54python {skill_path}/scripts/setup_experiment.py --list-evaluators55```5657## Built-in Evaluators5859| Name | Metric | Use Case |60| ------------------- | --------------------------- | ---------------------------------- |61| `benchmark_speed` | `p50_ms` (lower) | Function/API execution time |62| `benchmark_size` | `size_bytes` (lower) | File, bundle, Docker image size |63| `test_pass_rate` | `pass_rate` (higher) | Test suite pass percentage |64| `build_speed` | `build_seconds` (lower) | Build/compile/Docker build time |65| `memory_usage` | `peak_mb` (lower) | Peak memory during execution |66| `llm_judge_content` | `ctr_score` (higher) | Headlines, titles, descriptions |67| `llm_judge_prompt` | `quality_score` (higher) | System prompts, agent instructions |68| `llm_judge_copy` | `engagement_score` (higher) | Social posts, ad copy, emails |6970## After Setup7172Report to the user:7374- Experiment path and branch name75- Whether the eval command worked and the baseline metric76- Suggest: "Run `/ar:run {domain}/{name}` to start iterating, or `/ar:loop {domain}/{name}` for autonomous mode."7778> **Creator:** Engineering79> **License:** MIT80> **Source Repo:** `neekware/dojo-skills`81> **Source Bucket:** `engineering`82> **Original Path:** `engineering/autoresearch-agent/skills/setup`
neekware