Inference Workload Profile

Profile live inference traffic into a token/shape distribution artifact that drives profile-matched speculative-decoding draft training -- an analog of Fireworks FireOptimizer's "profile-driven customization" (the documented source of its higher draft hit-rate). Reads an OpenAI-style access JSONL, emits workload-profile.json (input/output length distributions, content-class mix, ISL/OSL bench shapes, and a spec-decode method recommendation), and hands off to inference-spec-decode-train via a hit-rate-matched corpus. The first phase of the adaptive spec-decode loop. Triggers on "profile my workload", "workload profile for spec-decode", "what draft should I train", "match the draft to my traffic", "adaptive speculative decoding", "fireoptimizer equivalent", "profile traffic for a draft model", or any combination of "profile / characterize / sample" with "workload / traffic / requests" and "spec-decode / draft / acceptance / hit-rate".

cfregly Updated 4 repo stars

File contents

cfregly/claude-perf-tune commit 56e055dcf6

Frequently asked questions

npx get-skill add cfregly/inference-workload-profile-2