Nl2sh Eval

nl2sh-eval

LLM-Supported Natural Language to Bash Translation — Westenfelder et al. (2025) (arXiv:2502.06858, 2025)

What this evaluates

This benchmark evaluates the ability of large language models to translate natural language instructions into executable Bash commands. It probes functional correctness by comparing model-generated commands against ground-truth commands using a functional equivalence heuristic that combines command execution with LLM-based output analysis.

Datasets

• NL2SH — total 41539; splits: train (40939), test (600); repo https://github.com/westenfelder/NL2SH
• InterCode-ALFA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly translated commands. Correctness is determined by the exec + mxbai-embed functional equivalence heuristic (FEH), which executes both the ground-truth and predicted commands, compares their stdout via cosine similarity of embeddings, and uses an LLM to verify functional equivalence with a 0.75 threshold.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for the functional equivalence heuristic evaluation, measuring how well the heuristic distinguishes equivalent from non-equivalent command pairs.

Input / output format

Input: Natural language instruction describing a shell task. For FEH evaluation, pairs of Bash commands (ground truth and predicted or rotated).

Output: A single Bash command string.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        out_pred = execute_command(pred)
        out_gold = execute_command(gold)
        sim = cosine_similarity(embed(out_pred), embed(out_gold))
        llm_ok = llm_evaluate_functional_equivalence(pred, gold, out_pred, out_gold)
        if sim >= 0.75 and llm_ok:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• The non-equivalent test pairs are created by arbitrarily rotating the third column of the dataset, which may not reflect realistic functional divergence or command syntax errors.
• All LLM evaluations use temperature=0 and seed=123, which eliminates stochasticity and may overestimate consistency compared to real-world usage.
• The 0.75 threshold for cosine similarity and LLM verdicts is fixed across all heuristics, potentially biasing precision/recall trade-offs for different model families.

Evidence (verbatim from paper)

All models are evaluated using version 0.3.6 of the InterCode-ALFA benchmark with the execution + mxbai-embed FEH. Accuracy is measured using the exec + mxbai-embed FEH.

Citation

@misc{westenfelder2025llmsupported,
  title={LLM-Supported Natural Language to Bash Translation},
  author={Westenfelder et al. (2025)},
  year={2025},
  note={arXiv:2502.06858}
}

• arXiv: 2502.06858