Inno Experiment Dev

inno-experiment-dev

Canonical Summary

Creates implementation plan, writes project code with judge feedback loop, and submits final experiment run. Use after code-survey in both Idea and Plan branches.

Trigger Rules

Use this skill when the user request matches its research workflow scope. Prefer the bundled resources instead of recreating templates or reference material. Keep outputs traceable to project files, citations, scripts, or upstream evidence.

Resource Use Rules

• Read from references/ only when the current task needs the extra detail.

Execution Contract

• Resolve every relative path from this skill directory first.
• Prefer inspection before mutation when invoking bundled scripts.
• If a required runtime, CLI, credential, or API is unavailable, explain the blocker and continue with the best manual fallback instead of silently skipping the step.
• Do not write generated artifacts back into the skill directory; save them inside the active project workspace.

Upstream Instructions

Inno Experiment Dev (Planning, Implementation, and Submission)

Merges the former inno-implementation-plan, inno-ml-dev-iteration, and the submit step of inno-experiment-submit-refine. Mirrors _create_implementation_plan (830-858), _implement_and_iterate (861-920), and the submit portion of _submit_and_refine_experiments (922-945) in run_infer_idea_ours.py.

Inputs

Plan mode additionally uses ideas and survey-specific prompt variants (build_plan_query_with_survey, build_iteration_query_for_plan, etc.).

Outputs

Cache Artifacts

Instructions

Phase 1: Create Implementation Plan

Mirrors _create_implementation_plan.

1. Optional pre-step (Idea mode only): If refining the idea for implementation clarity, call the idea refinement agent to produce refined_for_downstream with tensor interfaces and forward-pass sketch.

2. Build plan query:

   • Idea mode: plan_query = build_plan_query(survey_res, references, updated_prepare_res, code_survey_res, dataset_description) (see prompts/build_plan_query.md)
   • Plan mode: Use build_plan_query_with_survey(ideas, references, prepare_res, code_survey_res, dataset_description)

3. Call Coding Plan Agent with messages = [{"role": "user", "content": plan_query}].

   • The agent reviews codebases using tree / cat, then creates structured plans via plan_dataset, plan_training, plan_testing.
   • Calls case_resolved to merge plans.
   • Set plan_res = plan_messages[-1]["content"].
   • See references/coding_plan_agent.md for agent details.

4. Verify the plan has clear sections: dataset, model, training, evaluation, file layout.

Phase 2: Implement and Iterate

Mirrors _implement_and_iterate.

5. Initial implementation: Build ml_dev_query = build_ml_dev_query(survey_res, prepare_res, code_survey_res, plan_res, dataset_description, core_code, code_references) (see prompts/build_ml_dev_query.md). Use paths from instance.json: Experiment.core_code, Experiment.code_references (absolute in Dr. Claw–created projects; use as-is or resolve with project path if relative). Call ML Agent with messages = [{"role": "user", "content": ml_dev_query}]. Set ml_dev_res = ml_messages[-1]["content"].

   • See references/ml_agent_instructions.md for agent details.

6. Initial judge evaluation: Build judge_query = build_judge_query(survey_res, prepare_res, plan_res, ml_dev_res) (see prompts/build_judge_query.md). Call Judge Agent with input_messages = [{"role": "user", "content": judge_query}]. Set judge_res = judge_messages[-1]["content"].

   • See references/judge_agent_instructions.md for agent details.

7. Iteration loop (for i in 0..max_iter_times - 1): a. Build iteration_query = build_iteration_query(survey_res, prepare_res, code_survey_res, plan_res, ml_dev_res, judge_res, core_code, code_references) (see prompts/build_iteration_query.md). Use paths from instance.json (absolute in Dr. Claw–created projects; use as-is or resolve if relative). Plan mode uses build_iteration_query_for_plan. b. Append as user message to judge_messages. Call ML Agent with iter_times=i+1. Update ml_dev_res. c. Build judge_simple_query = build_judge_simple_query(survey_res, prepare_res, plan_res, ml_dev_res) (see prompts/build_judge_simple_query.md). Plan mode uses build_judge_simple_query_for_plan. d. Append as user message to judge_messages. Call Judge Agent with iter_times=i+1. Update judge_res. e. If "fully_correct": true in last message, break early.

8. Preserve judge_messages for the submit step and for downstream inno-experiment-analysis.

Phase 3: Submit Experiment

Mirrors the submit portion of _submit_and_refine_experiments.

9. Build submit query: submit_query = build_submit_query(survey_res, ml_dev_res, judge_res, core_code) (see prompts/build_submit_query.md). Resolve core_code from instance.Experiment.core_code. Plan mode uses build_submit_query_for_plan.

10. Append to judge_messages as user message. Call ML Agent with iter_times="submit".

    • The agent adjusts epochs (3-10), runs run_training_testing.py, ensures checkpoints are saved.
    • Set submit_res = judge_messages[-1]["content"].

11. If the implementation is not runnable, ML Agent calls case_not_resolved. Otherwise, case_resolved with statistical results and analysis.

Tool Mappings

All custom Python tools map to Claude Code built-in capabilities:

Checklist

• Optional idea refinement applied if desired (Idea mode).
• Correct build_plan_query variant used for Idea vs Plan mode.
• Coding Plan Agent called; plan_res has clear dataset/model/training/testing sections.
• ML Agent initial implementation completed; ml_dev_res recorded.
• Judge Agent initial evaluation completed; judge_res recorded.
• Iteration loop runs with correct prompt variants; early exit on fully_correct.
• judge_messages preserved across all phases.
• Submit query appended to judge_messages; ML Agent submission run completed.
• Final model checkpoint saved to Experiment/core_code/checkpoints/model_final.pth.
• Cache artifacts saved to Experiment/core_code/logs/: coding_plan_agent.json, machine_learning_agent.json, judge_agent.json, machine_learning_agent_iter_submit.json.

References

• run_infer_idea_ours.py: _create_implementation_plan (830-858), _implement_and_iterate (861-920), _submit_and_refine_experiments submit step (922-945)
• prompt_templates.py: build_plan_query (203-233), build_ml_dev_query (236-381), build_judge_query (384-417), build_iteration_query (420-468), build_judge_simple_query (471-494), build_submit_query (497-527)
• Agent definitions: plan_agent.py, ml_agent.py, judge_agent.py in inno/agents/inno_agent/