Explore Dnn Model

Explore DNN Model

Minimum Required Inputs (Hard Requirement)

To use this skill, the user must provide:

• A model checkpoint / model file(s) as a local file or directory path (it may be outside the workspace).

If the user provides only the checkpoint path (no model name, repo link, or source code), proceed by:

1. Attempting to identify the model name/family from the checkpoint file/dir itself (filenames, adjacent configs/README, embedded metadata, state_dict key patterns, etc.).
2. Searching for the implementation in the workspace and/or alongside the checkpoint directory (e.g., nearby Python packages, inference scripts, config files).
3. If still not found, using the best-guess model name/family to search online for the canonical implementation, then cloning the upstream source into tmp/<experiment-dir>/refs/ for investigation (prefer shallow clone; record URL + commit/tag used).

Goals

This skill has three goals:

1. Verify that the given DNN model can work (inference or training; default focus is inference) in the current Python environment of the workspace.
2. Determine how to use it (inference or training; default is inference) by reading the upstream source code and producing minimal, reproducible runs.
3. Produce two reports:
   • Experiment report (programmatic): generated from tmp/<experiment-dir>/outputs/ with minimal/no reasoning.
   • Stakeholder report (agent-written): generated by the agent from the experiment report + outputs/logs, with deeper analysis and recommendations.

The reports cover:

• Input and output contracts (formats, shapes, dtypes, preprocessing/postprocessing)
• Benchmarks and performance profiling (latency/throughput/memory, device details)
• User-provided metrics/targets (e.g., accuracy, mAP, IoU, F1, latency budget), and whether/how they are met

Before changing anything, detect how the environment is managed by checking for:

• pixi.toml and/or pyproject.toml (Pixi-managed project)
• .venv/ (venv-managed project)

Dependency Policy (Ask Once, Then Apply)

If any dependency is missing:

• Do not install it automatically without user confirmation.
• List the missing packages (and versions/constraints if known) and ask the developer how to proceed.
• Provide clear options, let the developer choose, then proceed with the chosen approach.
• Once the developer confirms an approach, apply it for all newly required packages (no need to ask approval per package).

Version Strategy

• First attempt: use the latest versions resolved by the selected package manager (pixi, pip, uv).
• If that fails (import/runtime errors, incompatibilities): fall back to the specific versions/constraints documented by the model’s upstream source code or docs.

Preferred Options (in order)

Pixi-managed env

• Ask the user to choose one:
  • Modify the current Pixi environment by adding deps to the relevant manifest (pixi.toml / pyproject.toml).
  • Create a new Pixi environment specifically to test this model.
• Then use pixi install/pixi run ... to execute.
• Prefer PyPI packages over conda-forge when both are available.
• Avoid direct pip install ... into the Pixi environment unless the developer explicitly requests it.

.venv-managed env Ask the user to choose one:

• Install deps via pip (or uv pip) into the current .venv.
• Create a new venv specifically for this model (keeps the repo venv clean).

Inputs to Collect (ask if missing)

• Model name and/or upstream repo link and/or source code path (optional but speeds up identification)
• Model task/modality if unclear (classification/detection/segmentation/embedding/audio/video/etc.)
• Checkpoint path (file/dir) and format (.pt, .pth, .onnx, .engine, etc.)
• Any known I/O contract details (expected resolution, channel order, normalization, label mapping), if the user has them
• CPU-only requirement (only if the user explicitly requests CPU-only)
• Optional: user-provided metrics/targets to evaluate (quality and/or performance)

Notes:

• Determine framework/runtime automatically from checkpoint type + upstream code/docs + what’s available in the current Python environment.
• If hardware is unspecified, default to using hardware acceleration when available (CUDA GPU, ROCm GPU, Apple MPS, etc.). Use CPU-only only if the user requested it.
• If unspecified, the default objective is to confirm the model runs end-to-end from input → output (prefer real inputs found in the workspace; synthesize as a fallback) and record end-to-end timing.

Core Workflow

0) Confirm artifacts and pick the target environment

• Confirm the minimum required inputs are present:
  • Checkpoint/model path is accessible locally (file/dir exists). It may be outside the workspace.
  • If model name/repo/source path is not provided, start by inferring it from the checkpoint and nearby files; if needed, locate it online and clone into tmp/<experiment-dir>/refs/.
• Detect environment type:
  • If both Pixi and .venv exist, ask the user which one should be treated as the “current” environment for this exploration.
• Device default:
  • If the user did not request CPU-only, use hardware acceleration when available (CUDA/ROCm/MPS/etc.).

1) Locate and read the upstream source code/docs

• First try to find the implementation locally:
  • Search the workspace and the checkpoint directory for source code, inference scripts, configs, and docs.
  • Prefer local source if it appears to be the canonical/official implementation for the checkpoint.
• If local source is not available or is clearly incomplete, use online search to find the canonical implementation:
  • Official GitHub repo, paper, model card, or vendor docs.
  • Check out the upstream repo under tmp/<experiment-dir>/refs/<repo-name> using a shallow clone (--depth=1), pinning a tag/commit when possible.
• Download/check out the relevant source code (pin a tag/commit when possible) and identify:
  • The exact inference entrypoints (scripts/modules), model class, preprocessing, postprocessing, and label mapping.
  • Any config files required to construct the model (YAML/JSON/TOML).
• Do not “guess” preprocessing/postprocessing: confirm from code and/or reference examples.

2) Derive required dependencies

Before running the model or changing the environment, determine the minimal dependencies required to run the model by using (in priority order):

• Upstream source code (setup files, requirements*.txt, pyproject.toml, import graph).
• Upstream docs/model card (pinned versions, known-good combos).
• Checkpoint type (e.g., .onnx implies ONNX Runtime; .pt/.pth implies PyTorch; .engine implies TensorRT).

Make a concise dependency list covering:

• Runtime/framework (e.g., torch, onnxruntime, opencv-python)
• Model-specific libs (e.g., ultralytics, timm, transformers, mmengine, etc.)
• Utility deps used by the official inference path (e.g., numpy, Pillow, pyyaml)
• Optional acceleration deps (CUDA/TensorRT) separated from the CPU baseline

3) Resolve missing dependencies (with user choice)

• Check whether each required dependency is available in the current environment.
• If anything is missing, ask the user which path to take:
  • Pixi: modify current manifest to add deps, or create a new Pixi env for this model.
  • Venv: install into current .venv, or create a new venv for this model.
• After the user confirms, apply the decision for all required packages (no per-package prompts).
• Use the Version Strategy above (latest first; fall back to pinned versions if needed).
• After dependency changes, run a quick smoke test:
  • Imports for the core runtime stack
  • Minimal “load model” path (without a full benchmark yet)

4) Ensure the checkpoint exists locally

• Do not download checkpoints automatically.
• Developers must provide checkpoints/model files (local file/dir paths).
• If the checkpoint is missing or only a URL is provided, ask the developer to download it and provide the local path.
• If the developer wants a conventional location, prefer checkpoints/ (gitignored).
• Record provenance in a short note (based on what the developer provides):
  • Claimed source URL(s) or repo, version/commit/tag (if known), file size, and (if feasible) SHA256.

5) Create an experiment workspace under tmp/

Default experiment directory:

<workspace>/tmp/<experiment-slug>-<time>

If the user specifies a different location/name, use the user-provided one instead.

Create the standard directory layout:

tmp/<experiment-dir>/
  README.md     # experiment intent + directory guide (keep updated)
  refs/         # checked-out upstream repos (use shallow clone for online checkouts)
    README.md
  scripts/      # throwaway but reproducible scripts (committed if useful)
    README.md
  inputs/       # downloaded/synthesized test inputs
    README.md
  outputs/      # artifacts + machine-readable stats (e.g., `stats.json`)
    README.md
  logs/         # logs (stdout/stderr, profiling traces, command transcripts)
    README.md
  reports/      # markdown notes: what was tried, params, results
    README.md
    figures/    # images embedded in reports
    experiment-report.md
    stakeholder-report.md

Shell safety note (avoid accidental directory names):

• Do not use bash brace expansion to create these folders (e.g., mkdir -p "$exp"/{refs,scripts,...}), because quoting/spacing mistakes can create literal directories like {refs,scripts,...}.

• Prefer a simple loop or explicit mkdir -p calls, for example:

  exp="tmp/<experiment-dir>"
  mkdir -p "$exp"
  for d in refs scripts inputs outputs logs reports reports/figures; do
    mkdir -p "$exp/$d"
  done
  

Conventions:

• Use relative paths from tmp/<experiment-dir> in scripts so the folder is movable.
• Keep scripts small and single-purpose (01_download_inputs.py, 10_infer.py, 20_visualize.py, …).
• Run Python via the selected environment manager:
  • Pixi: pixi run python ...
  • Venv: use the venv’s Python (avoid system Python)

README requirements:

• Create tmp/<experiment-dir>/README.md to describe:
  • The intention of the experiment (what model, what checkpoint, what question you’re answering)
  • How to reproduce (one-line pointer to the primary script(s))
  • A brief map of what each top-level subdir contains
• Each top-level subdir must have its own README.md that:
  • Describes what belongs in the folder
  • Notes any important changes (append a short “Changes” section as you iterate)

6) Collect or synthesize inputs

• First try to find suitable inputs already present in the workspace (e.g., under datasets/, downloads/, or other project-specific data dirs) based on what you learned from the checkpoint/source code (task, modality, expected resolution, file types).
• If no suitable inputs exist locally, synthesize minimal inputs that satisfy the model contract (e.g., generated images, random tensors saved in the expected container format, short synthetic video).
• Save all chosen/generated inputs under tmp/<experiment-dir>/inputs/.

7) Run minimal, traceable inference experiments (default: inference + end-to-end timing)

• Start with a single known-good example (from upstream repo) if available.
• Save every “input → output” mapping:
  • Inputs: the exact file(s) used + preprocessing parameters.
  • Outputs: raw model outputs + any decoded/visualized artifacts.
  • Command line + environment notes (device, precision, batch size).
• Measure end-to-end timing by default:
  • At minimum: one cold run + a small number of warm runs (record mean/median).
• Persist stats that will appear in the report:
  • For any timing/profiling/memory/throughput numbers you plan to put into the report, also write a JSON version under tmp/<experiment-dir>/outputs/ (e.g., outputs/stats.json).
• Capture logs by default:
  • Save stdout/stderr and command transcripts under tmp/<experiment-dir>/logs/.
• If the model is accessed via HTTP/gRPC, save request/response payloads (sanitized) under reports/ and/or outputs/.

7b) (Optional) Training sanity check

If the user asks to validate training (or if inference is insufficient to validate “works”):

• Start with a minimal configuration (single batch / tiny subset) to confirm the forward + backward pass runs.
• Record key configs (optimizer, LR, batch size, mixed precision) and any dataset assumptions.
• Do not run long trainings unless the user explicitly requests it.

8) Produce reports

8a) Ensure machine-readable report inputs exist (in outputs/)

Write/collect machine-readable files in tmp/<experiment-dir>/outputs/ that the report generator can consume, at minimum:

• stats.json (timing/throughput/memory/profile numbers)
• A JSON describing key parameters used (preprocess/postprocess/runtime thresholds)
• A JSON describing the I/O contract (input expectations + output structure)
• A JSON listing key artifacts produced (paths to representative inputs/outputs)

Keep these JSON files as the source of truth for anything that will appear as “final stats” in the experiment report.

8b) Generate reports/experiment-report.md programmatically

• Generate tmp/<experiment-dir>/reports/experiment-report.md by reading only tmp/<experiment-dir>/outputs/ (and optionally logs/ for pointers), with minimal/no reasoning.
• If images are part of the inputs/outputs, copy representative images into tmp/<experiment-dir>/reports/figures/ and embed them in the markdown via relative paths (e.g., figures/<name>.png).

8c) Write reports/stakeholder-report.md (agent-written)

• Read reports/experiment-report.md plus relevant outputs/ and logs/.
• Produce tmp/<experiment-dir>/reports/stakeholder-report.md with deeper analysis that requires reasoning:
  • Interpret results vs expectations/targets
  • Call out risks, assumptions, and failure modes
  • Recommend next experiments and concrete integration guidance (if requested)
  • Summarize “go/no-go” criteria and what remains unknown

Also include:

• Benchmark & profiling results:
  • CPU/GPU model, RAM/VRAM, OS, Python version, key library versions
  • Latency breakdown if possible (preprocess / model / postprocess)
  • Throughput (items/s) and peak memory/VRAM
• Stats JSON:
  • For any stats included in the report, ensure the same values exist in a JSON file under tmp/<experiment-dir>/outputs/ (e.g., outputs/stats.json).
• User metrics (if provided):
  • The metric definition + measurement method
  • Results on the chosen evaluation inputs
  • Any deltas vs the user’s targets and suggested next experiments

Guardrails

• Do not commit large checkpoints or huge outputs; keep them under gitignored paths (checkpoints/, tmp/).
• Respect upstream licenses; record the repo URL + commit/tag in reports/.
• Avoid modifying runtime code under src/ unless the user explicitly requests integration; keep exploration isolated to tmp/<experiment-dir>.