Text To Code Customization Eval

text-to-code-customization-eval

Exploring different approaches to customize language models for domain-specific text-to-code generation — Luís Freire, Fernanda A. Andaló, Nicki Skafte Detlefsen (arXiv:2603.16526, 2026)

What this evaluates

Evaluates the ability of customized small language models to generate correct and domain-aligned Python code. It probes functional correctness on general programming tasks versus specialized library APIs (Scikit-learn, OpenCV) under different customization strategies like few-shot prompting, RAG, and LoRA fine-tuning.

Datasets

• HumanEval — total ?; splits: test (-1)
• BCSk — total ?; splits: test (-1)
• BCCV — total ?; splits: test (-1)

Metrics

• Pass@1 (primary) — range: [0, 1]
  • The fraction of problems for which the generated code passes all provided test cases on the first attempt.
• Cosine Similarity (Sim.) — range: [0, 1]
  • The cosine similarity between the embedding of the generated code and the reference solution, measuring stylistic and structural alignment with the target dataset.

Input / output format

Input: Natural language task description, optionally augmented with few-shot examples or dynamically retrieved context snippets.

Output: A Python code snippet implementing the requested function or task.

Scoring recipe

def compute_pass_at_1(generated_code, test_cases):
    return 1.0 if run_tests(generated_code, test_cases) else 0.0

def compute_cosine_similarity(gen_code, ref_code):
    gen_emb = embed(gen_code)
    ref_emb = embed(ref_code)
    return cosine_similarity(gen_emb, ref_emb)

Common pitfalls

• RAG consistently improves similarity scores but does not guarantee higher Pass@1, as retrieved examples may introduce unnecessary operations.
• Few-shot prompting performance is highly sensitive to context window limits and can degrade with too many examples.
• Cosine similarity correlates with benchmark accuracy but measures stylistic alignment rather than functional correctness.

Evidence (verbatim from paper)

Performance is measured using benchmark-based metrics (Pass@1) and similarity metrics capturing alignment with the target datasets. For general Python tasks we use the HumanEval benchmark, while for library-specific tasks we use BigBenchCode subsets: BCSk for Scikit-learn and BCCV for OpenCV. Pass@1 measures benchmark correctness, and Sim. denotes cosine similarity to reference solutions.

Citation

@misc{freire2026exploring,
  title={Exploring different approaches to customize language models for domain-specific text-to-code generation},
  author={Luís Freire, Fernanda A. Andaló, Nicki Skafte Detlefsen},
  year={2026},
  note={arXiv:2603.16526}
}

• arXiv: 2603.16526