# Kdd Cup 24 Eval

> Evaluates instruction-tuned LLMs on e-commerce tasks across two Amazon KDD Cup'24 tracks. It measures model performance on development and official test sets to assess retrieval, ranking, and generation capabilities in a commercial setting. Use when the user wants to benchmark on Amazon KDD Cup'24, or asks about evaluating this task. Reports scores.

- Skill: `qhjqhj00/kdd-cup-24-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/kdd-cup-24-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/kdd-cup-24-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/kdd-cup-24-eval

---


# kdd-cup-24-eval

> EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization — Feng et al. (2024) (arXiv:2408.02970, 2024)

## What this evaluates

Evaluates instruction-tuned LLMs on e-commerce tasks across two Amazon KDD Cup'24 tracks. It measures model performance on development and official test sets to assess retrieval, ranking, and generation capabilities in a commercial setting.

## Datasets

- **Amazon KDD Cup'24** — total ?; splits: development (-1), official test (-1)

## Metrics

- `scores` **(primary)** — range: other
  - Official Amazon KDD Cup'24 scoring rules for Track 2 and Track 5. The exact mathematical formulation is not provided in the excerpt.

## Input / output format

**Input**: Instruction prompts covering five task types: generation, multiple-choice questions (MCQ), retrieval, ranking, and named entity recognition (NER).

**Output**: Model-generated responses or predictions corresponding to the five task types.

## Scoring recipe

```python
# Official KDD Cup'24 scoring for Track 2 & 5
# Not explicitly defined in the provided text
def score(predictions, gold):
    return official_kdd_cup_scoring(predictions, gold)
```

## Common pitfalls

- Quantizing models with out-of-domain datasets (e.g., C4) causes significant performance drops compared to in-domain data.
- Larger training sets do not always guarantee better performance; smaller sets can outperform in specific scenarios due to task trade-offs.

## Evidence (verbatim from paper)

> Table 2 demonstrates the performance of different models with the same training setting. Notably, Yi-1.5-34B achieved the highest scores across both Track 2 and 5 in both development and official test set. We also observed that models quantized using out-of-domain datasets C4 exhibited significant performance drops compared to those using in-domain sampled data.

## Citation

```bibtex
@misc{feng2024ecguide,
  title={EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2408.02970}
}
```

- arXiv: 2408.02970