qhjqhj00 cache-gen-eval
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Liu et al. (2023) (arXiv:2310.07240, 2023)
What this evaluates
Evaluates the efficiency and generation quality of a KV cache compression and streaming system for LLM serving. It measures how effectively the system reduces network bandwidth and time-to-first-token across varying context lengths, network conditions, and concurrent requests while maintaining task-specific accuracy, F1, or perplexity.
Datasets
- LongChat — total ?; splits: test (-1)
- TriviaQA — total ?; splits: test (-1)
- NarrativeQA — total ?; splits: test (-1)
- Wikitext — total ?; splits: test (-1)
Metrics
TTFT(primary) — range: seconds- Time from the arrival of the user query to the generation of the first token. Includes KV cache loading delay and new question prefetch delay.
KV cache size— range: bytes- Size of the KV cache after compression, used as a proxy for required network bandwidth.
Accuracy— range: [0, 1]- Percentage of generated answers that exactly include the ground-truth topic.
F1 score— range: [0, 1]- Measures the probability that the generated answer matches the ground-truth answer for QA tasks.
Perplexity— range: [0, ∞)- Exponentiated average negative log-likelihood of the next token. Used as a proxy for generation quality.
Input / output format
Input: Long text context (up to 32K tokens) provided as either raw text or pre-computed KV cache, paired with a user query or next-token prediction prompt.
Output: Generated text response (measured up to the first token for TTFT, and full response for quality metrics).
Scoring recipe
def evaluate(predictions, golds, dataset_type):
if dataset_type == 'LongChat':
return sum(1 for p, g in zip(predictions, golds) if g in p) / len(predictions)
elif dataset_type in ['TriviaQA', 'NarrativeQA']:
return sum(f1_score(p, g) for p, g in zip(predictions, golds)) / len(predictions)
elif dataset_type == 'Wikitext':
# Perplexity = exp(-1/N * sum(log(p(x_i))))
return perplexity(predictions, golds)
Common pitfalls
- TTFT only measures delay until the first token, not total generation time; comparing it to end-to-end latency is a common error.
- Perplexity is explicitly noted as a proxy metric that does not directly equate to text-generation quality, yet is often misinterpreted as such.
- Context compression baselines like H2O require query tensors unavailable during offline compression; the paper uses an idealized version, so real-world comparisons may differ.
Evidence (verbatim from paper)
Quality metrics: We measure generation quality using the standard metric of each dataset.
- Accuracy is used to evaluate the model's output on the LongChat dataset. The task predicts the first topic in the conversational history between the user and the LLM. The accuracy is defined as the percentage of generated answers that exactly includes the ground-truth topic.
- $F1$ score is used to evaluate the model's response in the TriviaQA and NarrativeQA datasets. It measures the probability that the generated answer matches the ground-truth answer of the question-answering task.
- Perplexity is used to evaluate the model's performance on the Wikitext dataset. The perplexity is defined as the exponentiated average negative log-likelihood of the next token [28, 41].
Citation
@misc{liu2023cachegen,
title={CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving},
author={Liu et al. (2023)},
year={2023},
note={arXiv:2310.07240}
}
- arXiv: 2310.07240