Results for “llm-serving”

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

Serve LLMs and VLMs with structured outputs, prefix caching, and high throughput using RadixAttention.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

Process textual and multimedia files with various LLM providers using the llm CLI. Supports both non-interactive and interactive modes with model selection, config persistence, and file input handling.

Deploy and serve LLMs in production with vLLM, Ollama, TGI, and llama.cpp, including quantization and GPU optimization.

Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

Summarize recorded video clips using the LVS microservice with a VLM fallback, producing a narrative summary with timestamped events.

Server-Sent Events for real-time server-to-client streaming. Express, Fastify, FastAPI, Spring WebFlux SSE implementations. Event streams, reconnection, and EventSource API. USE WHEN: user mentions "SSE", "Server-Sent Events", "EventSource", "event stream", "text/event-stream", "live feed", "streaming updates" DO NOT USE FOR: bidirectional communication - use `socket-io`; WebRTC - use `webrtc`; LLM streaming - use AI SDK skills

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production. Use when: langfuse, llm observability, llm tracing, prompt management, llm evaluation.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Provides expertise in Langfuse for LLM observability, including tracing, prompt management, evaluation, and integration with LangChain, LlamaIndex, and OpenAI.

Evaluates LLM performance using BLEU, ROUGE metrics and LLM-as-judge. Use for model testing.

Deploy Meta's LlamaGuard moderation model to filter LLM inputs and outputs across 6 safety categories using HuggingFace, vLLM, or FastAPI.

Build production ML systems with PyTorch 2.x, TensorFlow, and modern ML frameworks, including model serving, feature engineering, A/B testing, and monitoring.

Persistent memory CLI for LLM agents. Store facts, recall past knowledge, link related memories, manage lifecycle.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Implements and trains LLMs using Lightning AI's LitGPT with 20+ pretrained architectures (Llama, Gemma, Phi, Qwen, Mistral). Use when need clean model implementations, educational understanding of architectures, or production fine-tuning with LoRA/QLoRA. Single-file implementations, no abstraction layers.

Train, fine-tune, and deploy LLMs using LitGPT's clean implementations of 20+ architectures like Llama, Gemma, and Phi.

Selects the optimal LLM model and provider for each task based on complexity, cost budget, and capability requirements. Routes cheap tasks to Haiku/GPT-4o-mini and complex tasks to Sonnet/Opus/o1. Use when deciding which model to call, optimizing LLM costs, or building multi-model agent systems. Activate on "which model", "model selection", "route to model", "LLM cost", "model routing", "cheap vs expensive model". NOT for prompt engineering (use prompt-engineer), model fine-tuning, or training custom models.

Guides the creation of MCP servers that enable LLMs to interact with external services through well-designed tools.

Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

Data framework for building LLM applications with RAG. Specializes in document ingestion (300+ connectors), indexing, and querying. Features vector indices, query engines, agents, and multi-modal support. Use for document Q&A, chatbots, knowledge retrieval, or building RAG pipelines. Best for data-centric LLM applications.

AI 原生的潜在客户情报和外联流水线。用 agent 驱动的信号评分、共同关系人排名、暖场路径发现、来源语音建模和多渠道外联（邮件、LinkedIn、X），替代 Apollo、Clay 和 ZoomInfo。在用户想找到、评估并联系高价值联系人时使用。

Drive Soup (`soup-cli`), a CLI-first tool for fine-tuning and post-training LLMs with one YAML config and one command — SFT, DPO/GRPO/ORPO/SimPO/KTO, QLoRA/DoRA/LoRA+, layer streaming for 4-8 GB GPUs, eval-gated training, and serving. Use when the user wants to `soup init`/`soup train` a model, pick a training method or quantization scheme, estimate cost/memory before training, fine-tune on a small local GPU, migrate a config from Axolotl/LLaMA-Factory/Unsloth, or serve/merge/push a trained adapter. Triggers on: "soup-cli", "soup train", "soup init", "fine-tune an LLM locally", "QLoRA on a laptop GPU", "layer streaming", "soup advise", "soup autopilot", "DPO/GRPO/ORPO training", "merge LoRA adapter".

Expert in Langfuse - the open-source LLM observability platform. Covers tracing, prompt management, evaluation, datasets, and integration with LangChain, LlamaIndex, and OpenAI. Essential for debugging, monitoring, and improving LLM applications in production. Use when: langfuse, llm observability, llm tracing, prompt management, llm evaluation.