Results for “tensorrt-edge-llm”

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency on NVIDIA GPUs (A100/H100).

Integrates on-device and cloud machine learning into Flutter apps with TensorFlow Lite and Firebase ML Kit, covering image classification, object detection, OCR, face detection, and barcode scanning.

Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.

Import object detection models from HuggingFace or NVIDIA NGC into a DeepStream pipeline with automated ONNX download, TensorRT engine build, custom parser, multi-stream benchmark, and PDF report generation.

Train, evaluate, export, and run inference for monocular depth estimation models using Metric Depth Anything v2 or Relative Depth Anything architectures via the TAO toolkit.

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

Train, evaluate, distill, quantize, export, and run inference for RT-DETR object detection models using NVIDIA TAO.

Train, evaluate, export, and run inference for Masked Auto-Encoder (MAE) models for self-supervised pretraining and fine-tuning of visual representations.

Train, evaluate, export, quantize, and run inference for a TAO OneFormer model that performs panoptic, instance, and semantic segmentation using task-conditioned queries.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard.

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance using TensorBoard.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Fine-tune Gemma models locally using QLoRA, Unsloth, or TRL for SFT, DPO, and reward modeling, with dataset preparation and conversion to GGUF or LiteRT-LM.

Measures inference latency of binarized, 8-bit, and 32-bit convolutional layers on edge devices to evaluate the efficiency and speedup of the Larq Compute Engine framework compared to standard implementations.