Mp Rec Eval

mp-rec-eval

MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation — Hsia et al. (2023) (arXiv:2302.10872, 2023)

What this evaluates

Evaluates a hardware-software co-design framework for recommendation systems that dynamically switches between different embedding representations (table, DHE, hybrid) across heterogeneous hardware (CPU, GPU, IPU) to optimize throughput of correct predictions and model accuracy under strict latency constraints.

Datasets

• Kaggle — total ?; splits: test (10000)
• Terabyte — total ?; splits: test (10000)

Metrics

• Throughput of Correct Predictions (primary) — range: queries/sec
  • Product of model accuracy and system throughput (queries served per second). Measures the rate at which the system serves correct recommendations.
• Achievable model accuracy — range: percent
  • Percentage of correctly predicted recommendations out of total queries evaluated.
• SLA violation rate — range: percent
  • Percentage of queries whose execution latency exceeds the specified SLA latency target.

Input / output format

Input: Query sets with log-normal size distributions, accompanied by target QPS and SLA latency constraints.

Output: Dynamic routing decisions per query specifying the embedding representation (Table, DHE, Hybrid) and hardware platform (CPU, GPU, IPU) to execute.

Scoring recipe

def evaluate(queries, gold_labels, latencies, sla_target):
    correct = sum(1 for p, g in zip(queries, gold_labels) if p == g)
    accuracy = correct / len(gold_labels)
    throughput = len(gold_labels) / sum(latencies)
    throughput_correct = accuracy * throughput
    sla_violations = sum(1 for l in latencies if l > sla_target) / len(gold_labels)
    return {'throughput_correct_predictions': throughput_correct,
            'accuracy': accuracy,
            'sla_violation_rate': sla_violations}

Common pitfalls

• Assuming static representation-hardware mapping instead of dynamic switching, which ignores the framework's core optimization.
• Overlooking GPU offloading overheads for small queries, which makes CPU execution preferable in latency-sensitive scenarios.
• Ignoring the impact of query size distribution (log-normal) on performance scaling and hardware utilization.

Evidence (verbatim from paper)

MP-Rec achieves the highest model accuracy among all the embedding representations on both Kaggle and Terabyte datasets by using more accurate representations like DHE and hybrid (Section 6.2 - Insight 1). For Kaggle and Terabyte use-cases, MP-Rec conditionally improves achievable model accuracy by 0.19% and 0.22%, respectively (Table 2). ... MP-Rec improves throughputcorrectpredictions by 2.49× and 3.76× on Kaggle and Terabyte, respectively (Figure 10). ... At an SLA latency target of 10 ms, statically deploying embedding tables on CPUs will lead to 30.73% of queries violating SLA latency target.

Citation

@misc{hsia2023mprec,
  title={MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation},
  author={Hsia et al. (2023)},
  year={2023},
  note={arXiv:2302.10872}
}

• arXiv: 2302.10872