Kdd Cup 24 Eval

kdd-cup-24-eval

EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization — Feng et al. (2024) (arXiv:2408.02970, 2024)

What this evaluates

Evaluates instruction-tuned LLMs on e-commerce tasks across two Amazon KDD Cup'24 tracks. It measures model performance on development and official test sets to assess retrieval, ranking, and generation capabilities in a commercial setting.

Datasets

• Amazon KDD Cup'24 — total ?; splits: development (-1), official test (-1)

Metrics

• scores (primary) — range: other
  • Official Amazon KDD Cup'24 scoring rules for Track 2 and Track 5. The exact mathematical formulation is not provided in the excerpt.

Input / output format

Input: Instruction prompts covering five task types: generation, multiple-choice questions (MCQ), retrieval, ranking, and named entity recognition (NER).

Output: Model-generated responses or predictions corresponding to the five task types.

Scoring recipe

# Official KDD Cup'24 scoring for Track 2 & 5
# Not explicitly defined in the provided text
def score(predictions, gold):
    return official_kdd_cup_scoring(predictions, gold)

Common pitfalls

• Quantizing models with out-of-domain datasets (e.g., C4) causes significant performance drops compared to in-domain data.
• Larger training sets do not always guarantee better performance; smaller sets can outperform in specific scenarios due to task trade-offs.

Evidence (verbatim from paper)

Table 2 demonstrates the performance of different models with the same training setting. Notably, Yi-1.5-34B achieved the highest scores across both Track 2 and 5 in both development and official test set. We also observed that models quantized using out-of-domain datasets C4 exhibited significant performance drops compared to those using in-domain sampled data.

Citation

@misc{feng2024ecguide,
  title={EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantization},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2408.02970}
}

• arXiv: 2408.02970