Results for “llm-serving”

Recommends an inference runtime and memory-related launch flags for LLM/VLM workloads on NVIDIA Jetson devices, based on a live memory audit snapshot.

Deploy a trained model to serving with versioning, shadow or canary rollout, and a tested rollback path.

Deploy and serve LLMs in production with vLLM, Ollama, TGI, and llama.cpp, including quantization and GPU optimization.

You are an expert in vLLM, the high-throughput LLM serving engine. You help developers deploy open-source models (Llama, Mistral, Qwen, Phi, Gemma) with PagedAttention for efficient memory management, continuous batching, tensor parallelism for multi-GPU, OpenAI-compatible API, and quantization support — achieving 2-24x higher throughput than HuggingFace Transformers for production LLM serving.

Provides expertise in Langfuse for LLM observability, including tracing, prompt management, evaluation, and integration with LangChain, LlamaIndex, and OpenAI.

Deploy Meta's LlamaGuard moderation model to filter LLM inputs and outputs across 6 safety categories using HuggingFace, vLLM, or FastAPI.

Implements service mesh with Istio or Linkerd, covering traffic management, mTLS, circuit breakers, and observability for microservices on Kubernetes.

Provisions free LLM API keys from 20+ providers, health-checks existing keys, opens signup pages, validates new keys, and saves them to your project.