NVIDIA TensorRT-LLM

What is NVIDIA TensorRT-LLM?

NVIDIA TensorRT-LLM is a toolkit for running large language models and visual generation models faster on NVIDIA GPUs. It pairs optimised kernels for frequently used operations with a lean runtime and a Python framework that developers can customise and extend. The repository documents techniques such as speculative decoding, KV cache reuse and compression, disaggregated serving, expert parallelism for mixture-of-experts models, guided decoding and CUDA graph tuning. Recent notes cover optimisations for DeepSeek models on Blackwell GPUs, day-zero support for GPT-OSS models, a Jetson AGX Orin build, and diffusion-model support for visual generation. NVIDIA moved the project to a fully open-source model on GitHub in March 2025. It is free to use; teams that want a managed route can look at NVIDIA NIM microservices, which are part of the paid NVIDIA AI Enterprise offering.

Use cases

• Speed up LLM inference on NVIDIA GPUs
• Serve large models across multiple GPUs
• Apply speculative decoding to raise throughput
• Optimize DeepSeek models on Blackwell
• Run visual generation models efficiently
• Deploy LLMs on Jetson AGX Orin
• Benchmark and tune inference performance

Who it's for

• ML engineers
• Inference engineers
• AI platform teams
• Researchers

Why use NVIDIA TensorRT-LLM

• Raises tokens per second on NVIDIA hardware through kernels and runtime tuned for LLM workloads.
• Python-first design lets teams modify and extend the system without writing everything in C++.
• The project is open source on GitHub, so you can read, build and change the code.
• Publishes tech blogs and benchmarking guidance for models such as DeepSeek and GPT-OSS.
• Covers both text and visual generation models in a single toolkit.

NVIDIA TensorRT-LLM features

• Specialised kernels for common LLM operations
• Efficient runtime for LLM inference
• Pythonic framework that can be customised and extended
• Speculative decoding including n-gram methods
• KV cache reuse and compression
• Disaggregated serving of prefill and decode
• Expert parallelism for mixture-of-experts models
• Guided decoding combined with speculative decoding
• Diffusion model support for visual generation
• Documentation, architecture guide and performance overview

NVIDIA TensorRT-LLM integrations

• PyTorch
• CUDA
• Triton Inference Server
• Hugging Face
• NVIDIA NIM
• Amazon EKS

How to get started with NVIDIA TensorRT-LLM

1. Install TensorRT-LLM with a supported Python version and CUDA on an NVIDIA GPU machine.
2. Load a supported open-weights model through the Python API, using the quick-start examples.
3. Pick optimisations such as quantization, speculative decoding or KV cache reuse.
4. Run the optimized model locally or serve it, scaling across several GPUs or nodes if needed.
5. Measure results with the performance guide and adjust settings such as CUDA graph batch sizes.

NVIDIA TensorRT-LLM limitations

• Built for NVIDIA GPUs only
• Needs CUDA and a recent Python version
• Managed enterprise support comes through NVIDIA AI Enterprise, priced separately

NVIDIA TensorRT-LLM pricing

NVIDIA TensorRT-LLM is open source, so you can use and self-host it for free.

Prices can change and may vary by country or exclude tax. Check the official pricing page.

NVIDIA TensorRT-LLM alternatives

The main NVIDIA TensorRT-LLM alternatives listed on Debuly are vLLM, SGLang, llama.cpp, Ollama, Novita AI and NVIDIA Dynamo-Triton. The table compares pricing and platforms; order is A–Z.

1. 

   llama.cpp

   Open-source engine and desktop app for running LLMs locally, with a built-in OpenAI-compatible API server and web chat UI.

   Developer platformsCodingChat assistants
   Open sourceVisit llama.cpp (official site)
2. 

   Novita AI

   AI cloud with pay-per-token model APIs for 200+ models, dedicated GPU endpoints, agent sandboxes and GPU rentals.

   Developer platformsCodingAutomation and agents
   PaidVisit Novita AI (official site)
3. 

   NVIDIA Dynamo-Triton

   Open-source inference server, formerly Triton Inference Server, for serving models from TensorRT, PyTorch, ONNX and more in production.

   Developer platforms
   Open sourceVisit NVIDIA Dynamo-Triton (official site)
4. 

   Ollama

   Open-source app and CLI for running open AI models locally on macOS, Windows and Linux, with optional paid cloud models.

   Developer platformsCodingChat assistants
   Open sourceVisit Ollama (official site)
5. 

   SGLang

   Open-source serving framework for LLMs and multimodal models, with OpenAI-compatible endpoints and multi-GPU deployment.

   Developer platformsCoding
   Open sourceVisit SGLang (official site)
6. 

   vLLM

   Open-source engine for serving LLMs with PagedAttention, continuous batching and an OpenAI-compatible API.

   Developer platformsCoding
   Open sourceVisit vLLM (official site)

NVIDIA TensorRT-LLM FAQ

Sources & verification

Facts on this page come from the official sources below, each with the date we read it. Written by Debuly; the vendor did not review this page.

Image credits

• Logo: from avatars.githubusercontent.com · fetched 24 Oct 2026 · Vendor trademark, used only to identify the product; removed on request
• Cover: none collected yet, so a generated tile is shown.

Spot an error? Report an issue with this listing. We also remove a logo or image on the owner's request.