Results for “latency”

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).

Instrument LLM applications with Langfuse to trace, score, and monitor cost, quality, and latency across OpenAI and LangChain integrations.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Design multi-agent system architectures, generate tool schemas for Anthropic and OpenAI formats, and evaluate execution logs for cost, latency, and failure bottlenecks.

You are an expert in LLM observability and evaluation. You think in terms of traces, spans, and metrics. You know that LLM applications need monitoring just like traditional software - but with different dimensions (cost, quality, latency).

Trains, evaluates, exports, and runs inference for FastFoundationStereo (FFS) stereo depth estimation models, a distilled variant of FoundationStereo with lower latency.

Configures dynamic threshold alerting policies for Google Cloud Vertex AI Agent Platform agents, monitoring latency, error rates, and quality metrics using Terraform and PromQL.

You are an expert in LLM observability and evaluation. You think in terms of traces, spans, and metrics. You know that LLM applications need monitoring just like traditional software - but with different dimensions (cost, quality, latency).

Evaluates NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments. Use when setting up cosmos-policy for robot manipulation evaluation, running headless GPU evaluations with EGL rendering, or profiling inference latency on cluster or local GPU machines.

Build, deploy, evaluate, debug, and measure latency for a GPU-accelerated FastAPI service that detects whether operators perform assembly-line steps in order via event boundary detection and VLM classification.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

The layers trade timeliness against definitiveness.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Recommends an inference runtime and memory-related launch flags for LLM/VLM workloads on NVIDIA Jetson devices, based on a live memory audit snapshot.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Shares memory between agents at the representation level by compacting the orchestrator's KV cache for efficient worker handoff, reducing token costs without summarization or retrieval.

Debug LangChain and LangGraph agents by fetching execution traces from LangSmith. Analyze agent behavior, investigate errors, and review tool calls and performance metrics.

Caching strategies across the RAG stack. Semantic caching with GPTCache and LangChain, Redis-based embedding-similarity cache, cache key design, TTL/invalidation, partial caching (cache retrieval only), provider-native prompt caching (Anthropic, OpenAI), and hierarchical L1/L2 caches. USE WHEN: user mentions "semantic cache", "GPTCache", "LLM cache", "prompt caching", "Redis vector cache", "cache invalidation for RAG", "reduce LLM cost", "latency reduction LLM" DO NOT USE FOR: retrieval accuracy - use `rag-patterns`; groundedness checks - use `rag-guardrails`; incremental indexing - use `rag-production`

Instruments LLM applications with Langfuse for tracing, observability, and evaluation, covering setup, OpenAI and LangChain integrations, and best practices.

Expert guidance for Groq, the LLM inference platform that provides the fastest token generation speeds available, powered by custom LPU (Language Processing Unit) hardware. Helps developers integrate Groq's API for real-time AI applications where latency matters — chatbots, code completion, and streaming responses.

Design privacy-aware observability for AI agents using traces, spans, structured events, metrics, cost attribution, dashboards, alerts, and investigation workflows. Use when instrumenting an agent, debugging intermittent tool or model failures, defining service-level objectives, analyzing latency or spend, auditing agent decisions, or preparing production monitoring.

Instrument a shipped product's AI agents with tracing and observability so you can see what they did, why outputs happened, and what each run cost. Plain-language primer plus free-tier-first backend selection (Langfuse, Phoenix, LangSmith, Braintrust) and OpenTelemetry/OpenInference instrumentation. Load when the user asks to add observability, add tracing, instrument my agents, see what my agent is doing in production, set up Langfuse or Phoenix or LangSmith, debug why my agent gave a bad answer, or track LLM cost per request. Also fires when agent-system-architecture or setup-evaluation requires an observability plan for an agent-chain product. NOT for tracing the coding agent itself — that is run-trace. Precondition for runtime-learning-loop.

Detect prompt injections and jailbreak attempts in LLM applications using Meta's 86M parameter classifier. Filter user inputs, third-party data, and RAG documents with low latency and multilingual support.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.

Plans and coordinates the deployment of a model to Amazon SageMaker AI, selecting the appropriate pathway (real-time, serverless, async, batch, or Bedrock CMI) based on model type, traffic, latency, and cost constraints.

Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.