Recommend New Approaches

Recommend New Approaches

Synthesize training artifacts into a prioritized list of research directions, modeling ideas, and loss function modifications — sorted by expected impact.

Invocation

Arguments ($ARGUMENTS) are interpreted as:

• --out-dir DIR — directory containing run artifacts (default: ./reports/e2e-run); the skill reads any JSON files found here
• --run-id ID — run ID shared across orchestrated run
• --model PATH — explicit path to trained model or checkpoint (overrides auto-discovery)
• --eval-results PATH — path to check-eval.json (default: <out-dir>/check-eval.json)
• --train-results PATH — path to train-model.json (default: <out-dir>/train-model.json)
• --explain-results PATH — path to explain-model.json (default: <out-dir>/explain-model.json)
• --task-description TEXT — plain-language description of the task and business context (augments artifact data)
• --autoresearch — if set, attempt to use autoresearch for automated literature-grounded suggestions (requires autoresearch installed)

Target: $ARGUMENTS

Your responsibilities

1. Load and synthesize run artifacts

Collect all available signals from --out-dir or explicit paths:

• train-model.json — final metrics, hyperparameters, early stopping info, training duration, best checkpoint
• check-eval.json — task scores, baseline deltas, regressions, best gains
• explain-model.json — top features, PDP shapes, bias audit findings, unexpected patterns
• babysit-training.json — anomalies detected during training (NaN/OOM/divergence/stalled)
• check-dataset-quality.json — dataset profile, class imbalance, data quality findings
• review-target.json — task type, leakage risks, split strategy
• plan-experiment.json — original hypothesis, model family, HP search space

If no artifacts are found, ask for at least one of --eval-results or --train-results before proceeding. Work with partial data when only some artifacts are present — note which signals were unavailable.

2. Diagnose the current model's weaknesses

Before generating ideas, build a brief diagnostic picture:

• Performance gap: how far is the model from the promotion threshold or an ideal score?
• Training dynamics: did training converge smoothly, plateau early, or show instability?
• Evaluation patterns: which tasks/slices/classes underperform? Any regressions vs. baseline?
• Feature signal: are top features plausible? Any unexpected patterns or bias findings?
• Data constraints: class imbalance, small dataset, noisy labels, distribution shift signals?

This diagnostic shapes the recommendations — don't skip it even if only one artifact is available.

3. Generate recommendations

Produce 5–15 specific, actionable recommendations across these categories:

For each recommendation:

• Ground it in a specific signal from the artifacts (e.g., "eval_loss plateaued at step 2800 → suggest cosine annealing with restarts")
• Estimate expected impact (high / medium / low) based on the gap between current performance and where this idea could take it
• Estimate effort (high / medium / low) based on implementation complexity and compute cost
• Provide a concrete next_command when possible (e.g., a CLI flag change, a one-liner code edit, or a training command)

Sort all recommendations by expected_impact descending, breaking ties by effort ascending (high impact + low effort first).

4. Optionally run autoresearch

If --autoresearch is set, read the autoresearch repo and attempt to use it to run some of the recommendations you generated if they fit the autoresearch use case (iterations that take less than 5 mins but can run many of them)

uv run autoresearch "<task description derived from artifacts>" --max-papers 10 --output <out-dir>/autoresearch-results.json

Parse the results and merge any literature-grounded suggestions into the recommendation list, tagging them with "source": "autoresearch". If autoresearch is unavailable or fails, log a warning and continue with artifact-only recommendations.

Even without --autoresearch, you may draw on your knowledge of relevant ML literature to support recommendations — cite paper names or techniques by name where helpful.

5. Output a prioritized report

After generating recommendations, produce a readable summary:

Recommended Next Approaches
===========================
Model: <checkpoint or model ID>
Task: <task_type> — <primary_metric>=<score> (baseline delta: <delta>)
Diagnostic: <2-3 sentence summary of key weaknesses>

Top Recommendations (sorted by expected impact):

#1 [HIGH impact / LOW effort] — <category>: <idea title>
   Rationale: <why this matters given the artifacts>
   Next step: <concrete command or code change>

#2 [HIGH impact / MEDIUM effort] — <category>: <idea title>
   ...

Decision: CONDITIONAL
Confidence: medium

Use CONDITIONAL as the decision (these are forward-looking suggestions, not a hard gate). Use GO if the model already looks strong and only minor improvements remain.

JSON artifact

Write recommend-new-approaches.json to --out-dir (or ./ if invoked standalone) following the schema in ../../references/schemas.md. Use vocabulary from ../../references/vocabulary.md.

Key fields to populate:

• decision: CONDITIONAL in most cases (recommendations exist); GO if model is already near-optimal
• model, task_type, primary_metric, current_score, baseline_score, performance_gap
• diagnostic_summary: 2–3 sentence summary of key weaknesses
• recommendations: array sorted by expected_impact descending — see schema for full shape
• autoresearch_used: true / false
• autoresearch_path: path to autoresearch output file, or null
• artifacts_used: list of artifact paths that were loaded

Example session

/ml-skills:recommend-new-approaches --out-dir ./reports/run-001

Loading artifacts from ./reports/run-001:
  ✓ train-model.json — eval_loss=0.312, eval_f1=0.891 (early stopped at step 3200, patience=3)
  ✓ check-eval.json  — f1=0.891 vs. baseline 0.712 (+0.179); no regressions
  ✓ explain-model.json — top feature: transaction_amount_zscore; bias: disparate_impact=0.72 (failed)
  ✗ plan-experiment.json — not found, skipping

Diagnostic:
  Model trained well (F1 +0.179 over baseline) but stopped early at 32% of planned steps,
  suggesting the LR schedule may be too aggressive. Bias audit failed on disparate_impact (0.72 < 0.80),
  and the top feature shows heavy reliance on a single zscore — possible brittleness to distribution shift.

Recommended Next Approaches
===========================
Model: ./checkpoints/checkpoint-2800
Task: classification — eval_f1=0.891 (baseline delta: +0.179)

#1 [HIGH impact / LOW effort] — training: Cosine annealing with warm restarts
   Rationale: Early stopping triggered at step 3200 (patience=3) — LR plateau likely cause.
              Cosine restarts let training escape local minima without manual tuning.
   Next step: uv run train.py --lr_scheduler cosine_with_restarts --warmup_steps 200

#2 [HIGH impact / MEDIUM effort] — loss: Add fairness regularization term
   Rationale: disparate_impact=0.72 failed the 0.80 threshold. Adversarial debiasing or
              a fairness penalty on the loss can close the gap without major accuracy loss.
   Next step: Add FairLearn or AIF360 constraint wrapper around the training objective.

#3 [MEDIUM impact / LOW effort] — regularization: Increase early stopping patience to 7
   Rationale: patience=3 may be too tight — model stopped before plateau confirmed.
              A patience of 5–10 is common for tasks with noisy eval metrics.
   Next step: --early-stop-patience 7

...

Decision: CONDITIONAL
Confidence: medium

Artifact: ./reports/run-001/recommend-new-approaches.json

Additional resources

• autoresearch — automated literature-grounded research ideation
• check-eval — evaluation results (primary input)
• explain-model — feature importance and bias (supporting input)
• train-model — training artifacts (supporting input)
• ../../references/schemas.md — JSON artifact schema
• ../../references/vocabulary.md — canonical enum values