llama.cpp

What is llama.cpp?

llama.cpp is an open-source project for running large language models on your own computer, laptop or server. It loads models in the GGUF format, which packs the weights, tokenizer and metadata into one file, and thousands of ready-made GGUF models are on Hugging Face. The project ships a free Llama desktop app for Mac and Windows (a 1 MB download that lives in the menu bar or system tray), plus command-line tools: llama cli for chatting in a terminal and llama serve for an HTTP server with a web UI. The server speaks the OpenAI API and an Anthropic-style messages endpoint, supports streaming, tool calling, structured JSON output, vision, embeddings and reranking, and handles several users at once with continuous batching. The desktop app picks quantization and context size to suit your hardware, loads models on demand and unloads them after 5 idle minutes. llama.cpp is free and open source, with no API keys or usage bills, and it works offline.

Use cases

• Run open LLMs offline on a laptop
• Serve a local OpenAI-compatible API
• Point coding agents at a local model
• Generate embeddings for retrieval
• Rerank documents against a query
• Run vision models on local images
• Chat with a model in the terminal

Who it's for

• Developers
• ML engineers
• AI hobbyists
• Self-hosting teams

Why use llama.cpp

• Keeps prompts and files on your own machine, and the app works with no internet connection.
• Existing code written for the OpenAI API only needs its base URL changed, with no API keys or per-token bills.
• The desktop app chooses quantization, context size and batch size for your hardware, so setup takes a few clicks.
• Several apps can share one local server and one copy of each model instead of bundling their own engine.
• Every setting is still available in one plain-text file for people who want full control.

llama.cpp features

• Llama desktop app for Mac and Windows with a 1 MB download
• llama cli for terminal chat and llama serve for an HTTP API
• OpenAI-compatible endpoints plus an Anthropic-style /v1/messages route
• Built-in browser chat interface served by llama serve
• Streaming, tool calling and JSON-schema structured output
• Image and audio input for multimodal models
• Embedding and reranking endpoints for retrieval workflows
• Parallel request slots with continuous batching and prompt caching
• Speculative decoding with a small draft model
• Router mode for serving several models from one server
• Optional API key requirement and Docker image for servers

llama.cpp integrations

• Hugging Face
• OpenAI SDK
• Anthropic SDK
• Docker
• Tailscale

How to get started with llama.cpp

1. Install the Llama app with Homebrew or a download on Mac and Windows, or run the install script for the command-line version on Linux.
2. Pick a recommended model that fits your computer, or pass a Hugging Face GGUF repo to llama serve with the -hf flag.
3. Open the chat from the menu bar, or browse to localhost:9931 for the web UI.
4. Point your editor, coding agent or script at localhost:9931/v1 using any OpenAI SDK.
5. Adjust context size, GPU offload and parallel slots with flags or environment variables when you need to.

llama.cpp limitations

• Models run in the GGUF format, so other formats need converting
• Model size is bounded by the memory on your own machine
• Linux users install the command-line version rather than the desktop app

llama.cpp pricing

llama.cpp is open source, so you can use and self-host it for free.

Prices can change and may vary by country or exclude tax. Check the official pricing page.

llama.cpp alternatives

The main llama.cpp alternatives listed on Debuly are Ollama, LM Studio, vLLM, LocalAI, MLX and Jan. The table compares pricing and platforms; order is A–Z.

1. 

   Jan

   Free, open-source desktop AI assistant that runs models offline on your computer or connects to cloud models with your own keys.

   Chat assistantsDeveloper platforms
   Open sourceVisit Jan (official site)
2. 

   LM Studio

   Desktop app for running open language models on your own computer, now with the Bionic agent, a local API and optional US-hosted cloud models.

   Chat assistantsDeveloper platformsCodingProductivity
   FreemiumVisit LM Studio (official site)
3. 

   LocalAI

   MIT-licensed open-source runtime that serves local models through OpenAI- and Anthropic-compatible APIs for text, images, audio and agents.

   Developer platformsAutomation and agentsChat assistants
   Open sourceVisit LocalAI (official site)
4. 

   MLX

   Open-source array framework from Apple machine learning research for training and running models on Apple silicon.

   Developer platformsCoding
   Open sourceVisit MLX (official site)
5. 

   Ollama

   Open-source app and CLI for running open AI models locally on macOS, Windows and Linux, with optional paid cloud models.

   Developer platformsCodingChat assistants
   Open sourceVisit Ollama (official site)
6. 

   vLLM

   Open-source engine for serving LLMs with PagedAttention, continuous batching and an OpenAI-compatible API.

   Developer platformsCoding
   Open sourceVisit vLLM (official site)

llama.cpp FAQ

Sources & verification

Facts on this page come from the official sources below, each with the date we read it. Written by Debuly; the vendor did not review this page.

Image credits

• Logo: from llama.app · fetched 24 Oct 2026 · Vendor trademark, used only to identify the product; removed on request
• Cover: none collected yet, so a generated tile is shown.

Spot an error? Report an issue with this listing. We also remove a logo or image on the owner's request.