How do I install the Inference Optimization skill?

Run npx skillmds@latest add j4flmao/inference-optimization in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Inference Optimization skill do?

LLM Inference Optimization & MLOps It is listed under AI & ML on SkillMD.

Is Inference Optimization safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Inference Optimization?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Inference Optimization free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Inference Optimization?

j4flmao (@j4flmao) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Inference Optimization by j4flmao — Agent Skill

Name: Inference Optimization
Author: j4flmao

File contents

LLM Inference Optimization & MLOps

1. Skill Context

Focus: Optimizing the deployment of Large Language Models (LLMs) in production to maximize throughput, minimize VRAM usage, and reduce latency (Time-to-First-Token). Triggers: inference, vllm, paged-attention, quantization, awq, gptq, kv-cache.

2. The Inference Bottleneck: Memory Bandwidth

When running an LLM (e.g., Llama-3 70B), the math (Matrix Multiplication) is not the bottleneck. The bottleneck is loading the 70 Billion weights from the GPU's VRAM into the GPU's Compute Cores for every single token generated. Inference is fundamentally Memory-Bandwidth Bound.

3. The KV Cache Problem & PagedAttention

During token generation, the Transformer model must remember the calculations of all previous tokens. It stores these in the Key-Value (KV) Cache.

The Problem: In naive PyTorch/HuggingFace implementations, the KV Cache is pre-allocated contiguously in VRAM for the maximum possible sequence length (e.g., 8000 tokens). This wastes up to 80% of VRAM due to fragmentation. You quickly run Out Of Memory (OOM), drastically limiting how many concurrent users you can serve.
The Solution (vLLM & PagedAttention): Inspired by OS Virtual Memory, PagedAttention breaks the KV cache into fixed-size "blocks" (pages). The blocks do not need to be contiguous in VRAM. This eliminates fragmentation and allows the GPU to serve 3x-4x more concurrent users on the same hardware.

4. Quantization (Fitting 70B on limited GPUs)

A 70B parameter model in FP16 (16-bit float) requires ~140GB of VRAM just to load the weights (requiring 2x 80GB A100s). Quantization compresses the weights into lower precision without completely destroying the model's intelligence.

PTQ (Post-Training Quantization):
- GPTQ: Compresses weights to 4-bit integers. Reduces the 140GB model to ~35GB (fits on a single A6000). Highly optimized for batch processing.
- AWQ (Activation-Aware Weight Quantization): Also 4-bit, but smarter. It identifies the top 1% most "important" weights and keeps them in FP16, only quantizing the rest. Often yields better reasoning performance than GPTQ.
KV Cache Quantization: Beyond compressing the model weights, modern engines quantize the KV cache itself to FP8, saving even more VRAM during long context interactions.

5. Architectural Rule

Never use transformers.pipeline() for a production API serving multiple users. Always deploy a dedicated inference engine like vLLM, TGI (Text Generation Inference), or TensorRT-LLM to leverage continuous batching and PagedAttention.

---name: inference-optimizationdescription: LLM Inference Optimization & MLOps---# LLM Inference Optimization & MLOps## 1. Skill Context**Focus**: Optimizing the deployment of Large Language Models (LLMs) in production to maximize throughput, minimize VRAM usage, and reduce latency (Time-to-First-Token).**Triggers**: inference, vllm, paged-attention, quantization, awq, gptq, kv-cache.## 2. The Inference Bottleneck: Memory BandwidthWhen running an LLM (e.g., Llama-3 70B), the math (Matrix Multiplication) is not the bottleneck. The bottleneck is loading the 70 Billion weights from the GPU's VRAM into the GPU's Compute Cores for *every single token generated*. Inference is fundamentally **Memory-Bandwidth Bound**.## 3. The KV Cache Problem & PagedAttentionDuring token generation, the Transformer model must remember the calculations of all previous tokens. It stores these in the **Key-Value (KV) Cache**.- **The Problem**: In naive PyTorch/HuggingFace implementations, the KV Cache is pre-allocated contiguously in VRAM for the maximum possible sequence length (e.g., 8000 tokens). This wastes up to 80% of VRAM due to fragmentation. You quickly run Out Of Memory (OOM), drastically limiting how many concurrent users you can serve.- **The Solution (vLLM & PagedAttention)**: Inspired by OS Virtual Memory, PagedAttention breaks the KV cache into fixed-size "blocks" (pages). The blocks do not need to be contiguous in VRAM. This eliminates fragmentation and allows the GPU to serve 3x-4x more concurrent users on the same hardware.## 4. Quantization (Fitting 70B on limited GPUs)A 70B parameter model in FP16 (16-bit float) requires ~140GB of VRAM just to load the weights (requiring 2x 80GB A100s).**Quantization** compresses the weights into lower precision without completely destroying the model's intelligence.- **PTQ (Post-Training Quantization)**: - **GPTQ**: Compresses weights to 4-bit integers. Reduces the 140GB model to ~35GB (fits on a single A6000). Highly optimized for batch processing. - **AWQ (Activation-Aware Weight Quantization)**: Also 4-bit, but smarter. It identifies the top 1% most "important" weights and keeps them in FP16, only quantizing the rest. Often yields better reasoning performance than GPTQ.- **KV Cache Quantization**: Beyond compressing the model weights, modern engines quantize the KV cache itself to FP8, saving even more VRAM during long context interactions.## 5. Architectural RuleNever use `transformers.pipeline()` for a production API serving multiple users. Always deploy a dedicated inference engine like **vLLM**, **TGI (Text Generation Inference)**, or **TensorRT-LLM** to leverage continuous batching and PagedAttention.

Frequently asked questions

Inference Optimization