Inno Experiment Dev

Inno Experiment Dev (Planning, Implementation, and Submission)

Merges the former inno-implementation-plan, inno-ml-dev-iteration, and the submit step of inno-experiment-submit-refine. Mirrors _create_implementation_plan (830-858), _implement_and_iterate (861-920), and the submit portion of _submit_and_refine_experiments (922-945) in run_infer_idea_ours.py.

Inputs

Plan mode additionally uses ideas and survey-specific prompt variants (build_plan_query_with_survey, build_iteration_query_for_plan, etc.).

Outputs

Cache Artifacts

Instructions

Phase 1: Create Implementation Plan

Mirrors _create_implementation_plan.

1. Optional pre-step (Idea mode only): If refining the idea for implementation clarity, call the idea refinement agent to produce refined_for_downstream with tensor interfaces and forward-pass sketch.

2. Build plan query:

   • Idea mode: plan_query = build_plan_query(survey_res, references, updated_prepare_res, code_survey_res, dataset_description) (see prompts/build_plan_query.md)
   • Plan mode: Use build_plan_query_with_survey(ideas, references, prepare_res, code_survey_res, dataset_description)

3. Call Coding Plan Agent with messages = [{"role": "user", "content": plan_query}].

   • The agent reviews codebases using tree / cat, then creates structured plans via plan_dataset, plan_training, plan_testing.
   • Calls case_resolved to merge plans.
   • Set plan_res = plan_messages[-1]["content"].
   • See references/coding_plan_agent.md for agent details.

4. Verify the plan has clear sections: dataset, model, training, evaluation, file layout.

Phase 2: Implement and Iterate

Mirrors _implement_and_iterate.

5. Initial implementation: Build ml_dev_query = build_ml_dev_query(survey_res, prepare_res, code_survey_res, plan_res, dataset_description, core_code, code_references) (see prompts/build_ml_dev_query.md). Use paths from instance.json: Experiment.core_code, Experiment.code_references (absolute in Dr. Claw–created projects; use as-is or resolve with project path if relative). Call ML Agent with messages = [{"role": "user", "content": ml_dev_query}]. Set ml_dev_res = ml_messages[-1]["content"].

   • See references/ml_agent_instructions.md for agent details.

6. Initial judge evaluation: Build judge_query = build_judge_query(survey_res, prepare_res, plan_res, ml_dev_res) (see prompts/build_judge_query.md). Call Judge Agent with input_messages = [{"role": "user", "content": judge_query}]. Set judge_res = judge_messages[-1]["content"].

   • See references/judge_agent_instructions.md for agent details.

7. Iteration loop (for i in 0..max_iter_times - 1): a. Build iteration_query = build_iteration_query(survey_res, prepare_res, code_survey_res, plan_res, ml_dev_res, judge_res, core_code, code_references) (see prompts/build_iteration_query.md). Use paths from instance.json (absolute in Dr. Claw–created projects; use as-is or resolve if relative). Plan mode uses build_iteration_query_for_plan. b. Append as user message to judge_messages. Call ML Agent with iter_times=i+1. Update ml_dev_res. c. Build judge_simple_query = build_judge_simple_query(survey_res, prepare_res, plan_res, ml_dev_res) (see prompts/build_judge_simple_query.md). Plan mode uses build_judge_simple_query_for_plan. d. Append as user message to judge_messages. Call Judge Agent with iter_times=i+1. Update judge_res. e. If "fully_correct": true in last message, break early.

8. Preserve judge_messages for the submit step and for downstream inno-experiment-analysis.

Phase 3: Submit Experiment

Mirrors the submit portion of _submit_and_refine_experiments.

9. Build submit query: submit_query = build_submit_query(survey_res, ml_dev_res, judge_res, core_code) (see prompts/build_submit_query.md). Resolve core_code from instance.Experiment.core_code. Plan mode uses build_submit_query_for_plan.

10. Append to judge_messages as user message. Call ML Agent with iter_times="submit".

    • The agent adjusts epochs (3-10), runs run_training_testing.py, ensures checkpoints are saved.
    • Set submit_res = judge_messages[-1]["content"].

11. If the implementation is not runnable, ML Agent calls case_not_resolved. Otherwise, case_resolved with statistical results and analysis.

Tool Mappings

All custom Python tools map to Claude Code built-in capabilities:

Checklist

• Optional idea refinement applied if desired (Idea mode).
• Correct build_plan_query variant used for Idea vs Plan mode.
• Coding Plan Agent called; plan_res has clear dataset/model/training/testing sections.
• ML Agent initial implementation completed; ml_dev_res recorded.
• Judge Agent initial evaluation completed; judge_res recorded.
• Iteration loop runs with correct prompt variants; early exit on fully_correct.
• judge_messages preserved across all phases.
• Submit query appended to judge_messages; ML Agent submission run completed.
• Final model checkpoint saved to Experiment/core_code/checkpoints/model_final.pth.
• Cache artifacts saved to Experiment/core_code/logs/: coding_plan_agent.json, machine_learning_agent.json, judge_agent.json, machine_learning_agent_iter_submit.json.

References

• run_infer_idea_ours.py: _create_implementation_plan (830-858), _implement_and_iterate (861-920), _submit_and_refine_experiments submit step (922-945)
• prompt_templates.py: build_plan_query (203-233), build_ml_dev_query (236-381), build_judge_query (384-417), build_iteration_query (420-468), build_judge_simple_query (471-494), build_submit_query (497-527)
• Agent definitions: plan_agent.py, ml_agent.py, judge_agent.py in inno/agents/inno_agent/