Results for “llm-inference”
9 skills
orchestra-research Tensorrt LLM
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).
10.4k · bundle
nvidia Jetson Inference Mem Tune
Recommends an inference runtime and memory-related launch flags for LLM/VLM workloads on NVIDIA Jetson devices, based on a live memory audit snapshot.
2.2k · bundle
neuralblitz Llmops
Manages the lifecycle of large language models in production, covering model versioning, prompt management, inference optimization, and cost control.
1
More results
orchestra-research Serving Llms Vllm
Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.
10.4k · bundle
orchestra-research Llama Cpp
Run LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.
10.4k · bundle
orchestra-research Speculative Decoding
Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques for 1.5-3.6× speedup without quality loss.
10.4k · bundle
qhjqhj00 Reflex
Evaluates machine-generated log summaries without human-written references, using LLM judgment and dense embeddings to score relevance, informativeness, and coherence.
3
orchestra-research Knowledge Distillation
Compress large language models using knowledge distillation from teacher to student models, covering temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.
10.4k · bundle
orchestra-research Model Pruning
Compress large language models by 40-60% with minimal accuracy loss using one-shot pruning techniques like Wanda and SparseGPT, enabling faster inference and deployment on constrained hardware.
10.4k · bundle