Back to Glossary
Artificial Intelligence
TensorRT
TensorRT is NVIDIA's high-performance deep learning inference optimizer and runtime.
Definition
TensorRT is NVIDIA's high-performance deep learning inference optimizer and runtime that enables production-grade deployment of AI models. TensorRT optimizes models through: (1) Layer fusion — combining layers for efficiency, (2) Precision calibration — FP16, INT8 quantization, (3) Kernel auto-tuning — selecting optimal CUDA kernels, (4) Dynamic tensor memory — minimizing memory allocation. TensorRT-LLM extends this for LLM inference, supporting models like GPT, LLaMA, and BERT with 3-5x speedup vs native PyTorch. Harch Corp provides TensorRT-LLM environments on H100 GPUs.
Related Keywords
tensorrttensorrt llmnvidia inferenceai optimizationmodel optimization
Related Terms
AI Inference
AI inference is the process of using a trained model to make predictions on new data.
LLM Serving
LLM serving is the deployment of large language models for inference via APIs or applications.
NVIDIA H100 GPU
The NVIDIA H100 is a datacenter GPU built on the Hopper architecture, designed for AI training and inference at scale.