Results for “tensorrt-llm”

Train, evaluate, export, and run inference for monocular depth estimation models using Metric Depth Anything v2 or Relative Depth Anything architectures via the TAO toolkit.

Train, evaluate, distill, quantize, export, and run inference for RT-DETR object detection models using NVIDIA TAO.

Train, evaluate, export, quantize, and run inference for a TAO OneFormer model that performs panoptic, instance, and semantic segmentation using task-conditioned queries.

Trains, evaluates, exports, and runs inference for Siamese-network-based optical inspection models to detect manufacturing defects and quality issues in image pairs.

Integrates on-device and cloud machine learning into Flutter apps with TensorFlow Lite and Firebase ML Kit, covering image classification, object detection, OCR, face detection, and barcode scanning.

Deploy and serve LLMs with high throughput using vLLM's PagedAttention and continuous batching. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism for production inference.

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard.

Submit and manage TAO training, evaluation, and inference jobs on SLURM GPU clusters over SSH with sbatch/srun, Pyxis/Enroot containers, and Lustre-backed storage.

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

Train, evaluate, export, and run inference for CenterPose models used in 6-DoF object pose estimation with keypoint regression.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Trains, evaluates, exports, quantizes, and runs inference for a Mask Grounding DINO model for open-set instance segmentation guided by text prompts.

Train, evaluate, export, prune, and run inference for PointPillars 3D object detection models from LiDAR point clouds using NVIDIA TAO.

Train, evaluate, export, and run inference for Masked Auto-Encoder (MAE) models for self-supervised pretraining and fine-tuning of visual representations.

Launch distributed Megatron-LM training jobs on a SLURM cluster with a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules, container conventions, monitoring, and per-rank failure diagnosis.

Train, evaluate, distill, quantize, export, and run inference for PyTorch-based TAO image classification models with support for multiple backbones.

Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance using TensorBoard.

Build production ML systems with PyTorch 2.x, TensorFlow, and modern ML frameworks, including model serving, feature engineering, A/B testing, and monitoring.

Fine-tune Gemma models locally using QLoRA, Unsloth, or TRL for SFT, DPO, and reward modeling, with dataset preparation and conversion to GGUF or LiteRT-LM.

Compress large language models using knowledge distillation from teacher to student models, covering temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

Recommends an inference runtime and memory-related launch flags for LLM/VLM workloads on NVIDIA Jetson devices, based on a live memory audit snapshot.