adu2021 Overview
TimeBill solves the problem of unpredictable LLM inference latency in time-critical systems.
Core Technique
Three-Stage Pipeline:
# Stage 1: Response Length Prediction
length_predictor = fine_grained_response_length_model()
predicted_length = length_predictor.predict(input)
# Stage 2: Execution Time Estimation
ete = execution_time_estimator()
worst_case_time = ete.estimate(predicted_length)
# Stage 3: Adaptive KV Cache Eviction
cache_eviction_ratio = calculate_optimal_eviction(worst_case_time, time_budget)
When to Use
Use when: Robotics, autonomous driving, time-critical inference.
References
- Response length prediction
- FLOPs-based execution time modeling
- Adaptive KV cache management