Results for “llm-quantization”
15 skills
oyi77 llm-deployment
Deploy and serve LLMs in production with vLLM, Ollama, TGI, and llama.cpp, including quantization and GPU optimization.
10
orchestra-research serving-llms-vllm
Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.
10.4k · bundle
orchestra-research tensorrt-llm
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).
10.4k · bundle
More results
orchestra-research quantizing-models-bitsandbytes
Quantize LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss using bitsandbytes. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers.
10.4k · bundle
qhjqhj00 hqq-quantization
Quantize LLMs to 8/4/3/2/1-bit precision without calibration data, using multiple backends and HuggingFace/vLLM integration.
3 · bundle
majiayu000 hqq-quantization
Quantize large language models to 8/4/3/2/1-bit precision without calibration data, using multiple optimized backends and integrations with HuggingFace Transformers, vLLM, and PEFT/LoRA.
567 · bundle
orchestra-research hqq-quantization
Quantize large language models to 8/4/3/2/1-bit precision without calibration data, using multiple optimized backends for deployment with vLLM or HuggingFace Transformers.
10.4k · bundle
majiayu000 awq-quantization
Quantize large language models to 4-bit precision using activation-aware weight quantization, reducing memory footprint and speeding up inference with minimal accuracy loss.
567 · bundle
orchestra-research gptq
Quantize large language models to 4-bit with minimal accuracy loss using GPTQ, enabling deployment of 70B+ models on consumer GPUs with 4× memory reduction and 3-4× faster inference.
10.4k · bundle
orchestra-research gguf-quantization
Convert and quantize models to GGUF format for efficient CPU/GPU inference with llama.cpp, supporting 2-8 bit quantization and Apple Silicon acceleration.
10.4k · bundle
orchestra-research speculative-decoding
Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques for 1.5-3.6× speedup without quality loss.
10.4k · bundle
orchestra-research awq-quantization
Quantize large language models to 4-bit using activation-aware weight quantization, achieving ~3x speedup with minimal accuracy loss for deployment on limited GPU memory.
10.4k · bundle
orchestra-research knowledge-distillation
Compress large language models using knowledge distillation from teacher to student models, covering temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.
10.4k · bundle
orchestra-research implementing-llms-litgpt
Train, fine-tune, and deploy LLMs using LitGPT's clean implementations of 20+ architectures like Llama, Gemma, and Phi.
10.4k · bundle
orchestra-research llama-cpp
Run LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.
10.4k · bundle