Vision Arch Gen Eval

vision-arch-gen-eval

Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Design — Vysyaraju et al. (2025) (arXiv:2512.24120, 2025)

What this evaluates

Evaluates the classification performance of LLM-generated neural network architectures by training each for a single epoch on seven computer vision benchmarks. It measures Top-1 accuracy to assess architectural quality, while also tracking generation efficiency via hash validation speed and duplicate rejection rates.

Datasets

• MNIST — total 70000; splits: full (70000)
• CelebA-Gender — total 202000; splits: full (202000)
• CIFAR-10 — total 60000; splits: full (60000)
• CIFAR-100 — total 60000; splits: full (60000)
• ImageNette — total 13000; splits: full (13000)
• SVHN — total 600000; splits: full (600000)
• Places365 — total 1800000; splits: full (1800000)

Metrics

• Top-1 accuracy after 1 epoch (primary) — range: [0, 1]
  • Fraction of correctly classified samples out of the total number of samples after training the generated architecture for exactly one epoch.
• Hash computation time — range: milliseconds
  • Wall-clock time required to compute the whitespace-normalized hash for deduplication validation.
• Duplicates detected — range: count
  • Count of functionally identical architectures identified and rejected during the validation phase.

Input / output format

Input: Few-shot prompting examples (n=1 to 6) of neural network architectures alongside a target computer vision dataset specification.

Output: Generated neural network architecture code or configuration strings prefixed with 'alt-nn' variants.

Scoring recipe

for architecture in generated_architectures:
    for dataset in datasets:
        model = instantiate(architecture)
        train(model, dataset, epochs=1, optimizer='SGD', momentum=True)
        acc = compute_top1_accuracy(model, dataset)
        results[architecture][dataset] = acc

dataset_means = {ds: mean([r[ds] for r in results.values()]) for ds in datasets}
overall_mean = mean(dataset_means.values())
p_values = independent_t_test(variants, dataset_means)

Common pitfalls

• Training is strictly limited to 1 epoch for rapid validation, which may not reflect the true convergence or final performance of the generated architectures.
• Batch sizes vary across datasets (64-4096), potentially introducing variance in accuracy that is unrelated to architectural quality.
• Whitespace-normalized hash validation only eliminates functionally identical architectures with formatting differences, missing semantically equivalent but structurally distinct designs.

Evidence (verbatim from paper)

Primary Metric: Top-1 accuracy after 1 epoch Statistical Validation: Independent t-tests comparing variants within each dataset, dataset-balanced means for overall comparison

Training Configuration:

• • Epochs: 1 (for rapid validation)
• • Optimizer: SGD with momentum
• • Batch size: varies per dataset (typical: 64-4096)
• • Metric: Top-1 accuracy

Citation

@misc{vysyaraju2025enhancing,
  title={Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Design},
  author={Vysyaraju et al. (2025)},
  year={2025},
  note={arXiv:2512.24120}
}

• arXiv: 2512.24120