Squeez Tool Output Pruning Eval

squeez-tool-output-pruning-eval

Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents — Kovács (2026) (arXiv:2604.04979, 2026)

What this evaluates

Evaluates a model's ability to extract the smallest verbatim evidence block from a single tool observation given a focused query, while maximizing compression and correctly returning empty output for true negatives.

Datasets

• Squeez Benchmark — total 11477; splits: test (618); repo https://github.com/KRLabsOrg/squeez

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score computed between the predicted verbatim span and the gold span. Calculated as 2 * (precision * recall) / (precision + recall).
• Recall — range: [0, 1]
  • Token-level recall: fraction of gold tokens correctly predicted.
• Precision — range: [0, 1]
  • Token-level precision: fraction of predicted tokens that appear in the gold span.
• Exact — range: [0, 1]
  • Exact match accuracy: 1 if the predicted string exactly equals the gold string, else 0.
• Strict F1 — range: [0, 1]
  • Exact span match F1: 1 if prediction exactly matches gold, else 0. Often reported alongside token-level F1 to penalize partial matches.
• Compression — range: [0, 1]
  • Fraction of input tokens removed: 1 - (|prediction| / |gold|). Higher values indicate more aggressive pruning.

Input / output format

Input: A focused query string and a raw tool observation (e.g., git log, service log, docker logs, build output, kubectl status) provided as a single prompt.

Output: A verbatim subset of lines from the tool observation, or an empty string if no relevant content exists.

Scoring recipe

def compute_metrics(pred, gold):
    pred_tokens = pred.split()
    gold_tokens = gold.split()
    exact = 1.0 if pred == gold else 0.0
    common = Counter(pred_tokens) & Counter(gold_tokens)
    num_same = sum(common.values())
    prec = num_same / len(pred_tokens) if pred_tokens else 0.0
    rec = num_same / len(gold_tokens) if gold_tokens else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    compression = 1.0 - (len(pred_tokens) / len(gold_tokens)) if gold_tokens else 0.0
    return {'Exact': exact, 'Precision': prec, 'Recall': rec, 'F1': f1, 'Compression': compression}

Common pitfalls

• Heuristic baselines like BM25 fail because relevance depends on the query rather than lexical overlap alone.
• Models frequently select semantically adjacent but incorrect blocks in repetitive logs or Git history.
• Zero-shot models often generate explanatory text for true negatives instead of returning empty output.

Evidence (verbatim from paper)

Squeez-2B attains the highest recall among all systems while maintaining 92% compression. It outperforms the 18× larger Qwen 3.5 35B A3B by 11 recall points and the unfine-tuned 2B base by 33 points.

Citation

@misc{kovacs2026squeez,
  title={Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents},
  author={Kovács (2026)},
  year={2026},
  note={arXiv:2604.04979}
}

• arXiv: 2604.04979