Inference Spec Decode Service

Speculative decoding service workflow

Scope

This skill coordinates checked-in profiling and evidence tools with training, deployment, and evaluation commands supplied by the operator. The repository does not bundle a speculative decoding controller, cluster launcher, canary manifest, or draft training implementation.

Use this skill when a served model needs a draft matched to representative traffic and promotion must depend on measured serving results. Use inference-spec-decode-train for a single draft training job. Use inference-spec-decode-tune when the training recipe is already stable and several training configurations need a bounded comparison.

Required inputs

Stop before cluster work unless the operator provides all of these:

• A redacted traffic JSONL or a named workload shape set.
• The target checkpoint and current standing speculative config.
• Exact operator-owned commands or manifests for draft training, serving, and evaluation.
• A campaign matrix that names model identity, hardware, workload shape, concurrency, source revision, and trial count.
• An idle-node check appropriate for the operator's scheduler.
• An accuracy or acceptance floor and a latency objective.

Workflow

1. Create one identity

Call evidence_init. Use the returned run ID as the experiment ID, campaign ID, scheduler label, and join key. Store the operator-provided commands and source revisions in the evidence bundle before execution.

2. Profile representative traffic

From the repository root, run the checked-in profiler:

python3 plugins/profile-and-optimize/skills/inference-workload-profile/tools/workload-profile.py \
  --in <redacted-traffic.jsonl> \
  --out <bundle>/workload-profile.json

For the bundled synthetic AA shapes, replace --in with --aa-shapes. Review the input and output length distributions, concurrency, content mix, and recommended method. A profile from a different traffic regime is not a valid training input.

3. Build the matched corpus

Use the checked-in corpus converter:

python3 plugins/profile-and-optimize/skills/inference-workload-profile/tools/profile-to-corpus.py \
  --profile <bundle>/workload-profile.json \
  --traffic <redacted-traffic.jsonl> \
  --out <bundle>/corpus.jsonl

Verify that the target tokenizer and chat template render a sample correctly. Record the corpus hash. Do not retain unredacted production prompts in a public bundle.

4. Verify capacity, then train

Use scheduler-native read-only checks. For example, use sinfo and squeue for Slurm, or kubectl get nodes and kubectl get pods -A for Kubernetes. Capture the selected node and any existing workload before submitting. Never infer idleness from a node label alone.

Hand the corpus and target checkpoint to inference-spec-decode-train. The operator-provided trainer remains the source of truth. Capture its exact argv, stdout, stderr, exit code, and output hashes. A failed capture-correctness probe or non-finite loss stops the workflow.

5. Run a controlled serving comparison

Prepare one canonical matrix YAML with cell_id entries for the standing and candidate configs. Each vLLM cell must include its top-level identity fields and a vllm_sweep block with operator-provided serve_cmd and bench_cmd.

Initialize the campaign with the required config:

{
  "args": [
    "--config", "<matrix.yaml>",
    "--experiment-id", "<run-id>",
    "--family", "<family>",
    "--evidence-bundle", "<bundle>"
  ]
}

Dry-run perf_tune_report_campaign_run with --config, --campaign, and --dry-run in args. Inspect placement, deployment changes, workload shape, warmup, and cleanup. The MCP dry-run does not require a mutation acknowledgement.

For the live call, pass the structured field i_understand_this_mutates_cluster: true. Never place an acknowledgement flag inside raw args.

6. Gate the result

Measure in-engine accepted tokens and a user-facing latency metric such as TPOT or ITL. Run at least three matched trials on the same node and under the same graph mode. Record separate directional baselines:

• Acceptance and throughput use --direction higher-is-better.
• TPOT, ITL, and TTFT use --direction lower-is-better.

Promote only when the candidate passes the acceptance floor and improves the declared latency objective outside the noise bound. Training accuracy is a proxy and cannot authorize promotion.

7. Publish only reviewed evidence

Aggregate and render the campaign. Keep incomplete results marked DRAFT. Use a dry-run of perf_tune_report_publish_to_lake first. A live external publish requires i_understand_this_publishes_externally: true for that call.

Safety rules

• Never preempt a standing serving workload to create training capacity.
• Give every experiment object a unique name and experiment=<run-id> label.
• Delete only objects selected by that label. Preserve persistent volumes until outputs and hashes are captured.
• Keep the standing speculative config until the controlled comparison wins.
• Treat controller deployment and automatic promotion as out of scope. They require a separate reviewed implementation.

Reporting

Every number must include model, hardware, quantization, parallelism, serving config, workload shape, concurrency, warm or cold regime, source revision, trial count, and named baseline. End with the next measured lever and the gate that can prove or reject it.