aibot88 Research Workflow
Purpose
Optimize moai-adk components (skills, agents, rules, config) through iterative binary eval experimentation.
Data Locations
| Data | Location |
|---|---|
| Eval suites | .moai/research/evals/{type}/{name}.eval.yaml |
| Baselines | .moai/research/baselines/{name}.baseline.json |
| Experiments | .moai/research/experiments/{name}/exp-NNN.json |
| Changelogs | .moai/research/experiments/{name}/changelog.md |
| Observations | .moai/research/observations/ |
| Dashboard | moai research status (CLI) |
Eval Suite Schema
target:
path: .claude/skills/moai-lang-go/SKILL.md
type: skill
test_inputs:
- name: scenario-name
prompt: "Test prompt for evaluation"
evals:
- name: criterion-name
question: "Binary yes/no question?"
pass: "What yes looks like"
fail: "What triggers no"
weight: must_pass # or nice_to_have
settings:
runs_per_experiment: 3
max_experiments: 20
pass_threshold: 0.80
target_score: 0.95
Safety Layers
- FrozenGuard: Constitution files cannot be modified
- Worktree Sandbox: All experiments in isolated worktrees
- Canary Regression: Proposed changes tested against baselines
- Rate Limiter: Max experiments per session/week
- Human Approval: Required before merging to main
Common Rationalizations
| Rationalization | Reality |
|---|---|
| "The experiment results are obvious, I do not need a baseline" | Without a baseline, you cannot distinguish improvement from noise. Always measure before and after. |
| "This change is clearly better, I can skip the canary check" | Canary checks catch regressions the author cannot predict. Clear-to-author is not clear-to-system. |
| "I will merge the research branch now and validate later" | Unvalidated merges bypass every safety layer. Validate in the worktree before proposing a merge. |
| "The rate limiter is too restrictive for rapid experimentation" | Rate limits prevent runaway self-modification. If you need more experiments, batch them. |
| "I can modify the constitution to improve it" | Constitution files are FROZEN. Propose amendments through the human approval channel. |
Red Flags
- Experiment branch merged without canary regression results
- Baseline measurement missing from the experiment report
- Multiple experiments run in the same session exceeding the rate limit
- Worktree not used for experimental changes (changes made in main workspace)
- Constitution or frozen-zone file modified during a research session
Verification
- Experiment conducted in an isolated worktree (show worktree path)
- Baseline measurement recorded before any changes
- Canary regression check passed (show before/after comparison)
- Rate limit not exceeded (check experiment count in session)
- Human approval obtained before merge proposal
- No frozen-zone files were modified (verify with git diff --name-only)