Rospr Eval

rospr-eval

Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt — Ye et al. (2022) (arXiv:2210.03029, 2022)

What this evaluates

Evaluates the zero-shot generalization capability of instruction-tuned language models by retrieving and applying task-specific soft prompt embeddings at inference time to adapt to unseen tasks.

Datasets

• BIG-bench — total ?; splits: test (-1)
• SuperGLUE/HellaSwag/StoryCloze/WiC suite — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answer choices or generated outputs matching the gold labels across all tasks.

Input / output format

Input: Hard prompt template concatenated with the input instance, followed by a retrieved soft prompt embedding vector.

Output: Predicted answer choice or text generation corresponding to the hard prompt's expected output format.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize(pred) == normalize(gold):
        correct += 1
    total += 1
return (correct / total) * 100

Common pitfalls

• Retrieval is performed on a randomly sampled subset of target task instances (Q) rather than the full test set, which may introduce variance.
• Soft prompts are trained on source tasks and retrieved via dense embedding similarity, which may not perfectly align with target task semantics or answer choice formats.

Evidence (verbatim from paper)

improving performance on 10 out of 11 tasks and boosting BIG-bench mean accuracy by 2.39% with only 0.007% additional parameters.

Citation

@misc{ye2022rospr,
  title={Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt},
  author={Ye et al. (2022)},
  year={2022},
  note={arXiv:2210.03029}
}

• arXiv: 2210.03029