Results for “vlm-serving”

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

vLLM: high-throughput LLM serving, OpenAI API, quantization.

Summarize recorded video clips using the LVS microservice with a VLM fallback, producing a narrative summary with timestamped events.

Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Deploy a standalone RT-VLM dense-captioning microservice and exercise its REST API endpoints for file upload, caption generation, streaming, chat completions, and Kafka integration.

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

Ask visual questions about video clips using a VSS agent's video_understanding tool, requiring a fresh look at frames rather than prior metadata or search results.

Deploy and serve LLMs in production with vLLM, Ollama, TGI, and llama.cpp, including quantization and GPU optimization.

Process textual and multimedia files with various LLM providers using the llm CLI. Supports both non-interactive and interactive modes with model selection, config persistence, and file input handling.

Deploy and operate the RTVI-CV-3D microservice for multi-camera 3D detection and tracking, supporting sample datasets, custom videos, and RTSP streams.

Build, deploy, evaluate, debug, and measure latency for a GPU-accelerated FastAPI service that detects whether operators perform assembly-line steps in order via event boundary detection and VLM classification.

Deploy and operate the VSS 3.2 GA RT-Embed Video Embedding microservice using Docker Compose, covering GPU prerequisites, REST API usage for file uploads, text/video embeddings, live RTSP streams, Redis/Kafka/OTel integration, and troubleshooting.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Queries video analytics incidents, alerts, metrics, and sensor data from Elasticsearch via the VA-MCP server.

Deploy, debug, and operate the RTVI-CV 2D detection/tracking microservice and call its REST API for stream management, health checks, and metrics.

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).

Enables visual instruction tuning and image-based conversations using open-source vision-language models. Supports multi-turn image chat, visual question answering, and image understanding tasks.

SNLI-VE: Visual Entailment Dataset

Serve LLMs and VLMs with structured outputs, prefix caching, and high throughput using RadixAttention.

Server-Sent Events for real-time server-to-client streaming. Express, Fastify, FastAPI, Spring WebFlux SSE implementations. Event streams, reconnection, and EventSource API. USE WHEN: user mentions "SSE", "Server-Sent Events", "EventSource", "event stream", "text/event-stream", "live feed", "streaming updates" DO NOT USE FOR: bidirectional communication - use `socket-io`; WebRTC - use `webrtc`; LLM streaming - use AI SDK skills

飞书视频会议：查询进行中的会议列表（含会议 ID）、读取会中实时内容（发言、聊天、共享等）、发送会中消息，以及搜索历史会议、查询会议纪要（总结/待办/章节/逐字稿）和参会人快照。本技能不覆盖 Agent 真实入会/离会；查询未来日程走 lark-calendar。

Runs the open-source LLaVA vision-language model for image understanding, captioning, visual question answering, and multi-turn image conversations, including setup, inference, and training guidance.

飞书视频会议：搜索历史会议记录、查询会议纪要（总结/待办/章节/逐字稿）、查询参会人快照。当用户查询已结束的会议、获取会议产物（纪要/妙记）、查看参会人时使用；查询未来日程走 lark-calendar。不负责：Agent 真实入会/离会、会中实时事件（走 lark-vc-agent）。

NVIDIA's runtime safety framework for LLM applications. Features jailbreak detection, input/output validation, fact-checking, hallucination detection, PII filtering, toxicity detection. Uses Colang 2.0 DSL for programmable rails. Production-ready, runs on T4 GPU.

Drive Soup (`soup-cli`), a CLI-first tool for fine-tuning and post-training LLMs with one YAML config and one command — SFT, DPO/GRPO/ORPO/SimPO/KTO, QLoRA/DoRA/LoRA+, layer streaming for 4-8 GB GPUs, eval-gated training, and serving. Use when the user wants to `soup init`/`soup train` a model, pick a training method or quantization scheme, estimate cost/memory before training, fine-tune on a small local GPU, migrate a config from Axolotl/LLaMA-Factory/Unsloth, or serve/merge/push a trained adapter. Triggers on: "soup-cli", "soup train", "soup init", "fine-tune an LLM locally", "QLoRA on a laptop GPU", "layer streaming", "soup advise", "soup autopilot", "DPO/GRPO/ORPO training", "merge LoRA adapter".

Search the web with LLM-optimized results via the Tavily CLI, returning relevant snippets, relevance scores, and metadata. Supports domain filtering, time ranges, and multiple search depths.

Call 100+ LLMs through a single OpenAI-compatible interface with LiteLLM — use completion/acompletion/embedding with any provider (Anthropic, OpenAI, Google, Groq, Ollama, etc.), run a proxy server for team rate-limiting and cost tracking, load-balance across providers.