# Hscodecomp Eval

> This benchmark evaluates deep search agents' ability to perform multi-hop reasoning across hierarchical tariff rules to predict 10-digit Harmonized System Codes (HSCode) from noisy product descriptions and images. It probes rule-based reasoning, agentic knowledge utilization, and handling of vague or implicit classification logic. Use when the user wants to benchmark on HSCodeComp, or asks about evaluating this task. Reports 10-digit accuracy.

- Skill: `qhjqhj00/hscodecomp-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/hscodecomp-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/hscodecomp-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/hscodecomp-eval

---


# hscodecomp-eval

> HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application — Yang et al. (2025) (arXiv:2510.19631, 2025)

## What this evaluates

This benchmark evaluates deep search agents' ability to perform multi-hop reasoning across hierarchical tariff rules to predict 10-digit Harmonized System Codes (HSCode) from noisy product descriptions and images. It probes rule-based reasoning, agentic knowledge utilization, and handling of vague or implicit classification logic.

## Datasets

- **HSCodeComp** — total ?; splits: test (-1)

## Metrics

- `10-digit accuracy` **(primary)** — range: percent
  - Calculated as the percentage of test instances where the model's predicted 10-digit HSCode exactly matches the ground truth code.

## Input / output format

**Input**: Textual product description and optionally product images.

**Output**: A single 10-digit HSCode enclosed in \boxed{...} format.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_code = pred.replace('\\boxed{', '').replace('}', '').strip()
        if pred_code == gold.strip():
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Including hierarchical decision rules (DR) in the prompt actually decreases agent accuracy, contrary to intuition.
- Full webpage visiting tools overwhelm agents with irrelevant content, reducing accuracy compared to using search engine snippets.
- Closed-source agents were evaluated on a small manual subset (49 examples), which may not be directly comparable to the full automated test set.

## Evidence (verbatim from paper)

> All systems produce standardized outputs: a single HSCode in \boxed{...} format. ... The best baseline, SmolAgent (GPT-5 VLM version), achieves only 46.83% 10-digit accuracy, which remains substantially below the 95% accuracy achieved by experienced human experts.

## Citation

```bibtex
@misc{yang2025hscodecomp,
  title={HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2510.19631}
}
```

- arXiv: 2510.19631