How to run open-source LLMs locally: a step-by-step guide

By the end of this guide you will be able to run an open-source large language model on your own computer, chat with it, and call it from your own code through a local API, with no prompts leaving your machine. The software that does this is a local model runner: an app or command-line tool that downloads open model files and serves them on your hardware.

To run an LLM locally you need to match three things: your hardware, the size of the model, and the kind of runner that suits how you work. This guide covers that decision and a first setup, and compares six tools from our Developer platforms category.

Key takeaways

• Check your memory and chip first: local speed and the largest model you can load depend on your own hardware.
• The main decision is the runner: a desktop app with a chat window, a command-line tool and API, a prebuilt multi-service stack, or a small runtime for edge devices.
• Most runners expose an OpenAI-compatible local API, so existing code usually needs only a changed base URL.
• Local use keeps data on your device, but downloaded models still have licences and can still be wrong, so check output that matters.

What you need before you start

• A computer with enough memory. Larger models need more RAM or video memory. Some listings warn that big models need a lot of memory, so check the model's size before downloading.
• Free disk space. Model files are large.
• An operating system the tool supports. Ollama and LM Studio list macOS, Windows and Linux, Rapid-MLX needs an Apple Silicon Mac, and Harbor lists macOS and Linux.
• Docker, for some tools. Harbor needs Docker for its container-based services.
• Time. A first model can be chatting with you in under an hour, with most of that spent downloading.
• Difficulty. Easy with a desktop app, moderate with the command line, and harder with a multi-service stack.

Which kind of AI tool fits this job

Choose by how you want to interact with the model.

A desktop app for chatting and trying models. LM Studio is a desktop application for downloading and running open models on macOS, Windows and Linux. You can chat, ingest documents and use MCP servers through the interface, and it also runs a local OpenAI-compatible server and a CLI. Data stays on your device when you use models locally.

A command-line tool and API for developers. Ollama provides a desktop app, CLI and REST API, with official Python and JavaScript libraries and OpenAI-compatible and Anthropic-compatible clients. Its listing states that running models on your own hardware is always unlimited, and it also offers paid cloud models for when you lack a local GPU.

Apple Silicon specialists. Rapid-MLX is a free local server and Mac app built on MLX. It exposes OpenAI-compatible and Anthropic-compatible endpoints and lets several clients share one Mac.

A prebuilt stack of local services. Harbor starts a chosen set of services through Docker Compose and wires them together, such as inference backends (Ollama, llama.cpp, vLLM), chat frontends like Open WebUI, and extras for web search, voice chat and image generation.

A small portable runtime. LlamaEdge is a Rust and WebAssembly runtime that serves local models through an OpenAI-compatible API on CPUs, GPUs and edge devices, with GGUF model files.

An application layer on top of a local server. PrivateGPT does not run models itself. It sits between your app and a self-hosted inference server such as Ollama or LM Studio, and adds file ingestion, retrieval with citations, tools and MCP connectors.

Step 1: Check your hardware and set a goal

You finish this step knowing how large a model your machine can reasonably handle and what you want it to do.

1. Find your total memory, your chip or GPU, and your free disk space.
2. Decide the job: general chat, coding help, summarising documents, or an API for an app.
3. Check the runner supports your system. For example, Rapid-MLX lists Apple Silicon (M1 or newer) with macOS 14 or later, and Intel Macs are not supported.
4. Start with a smaller model than you think you need. You can move up later.

Check it worked by writing down your memory, your operating system and the job in one line.

Step 2: Choose a runner

Pick the simplest tool that covers your job.

1. For a chat window in a desktop app, start with LM Studio.
2. For scripts and applications, start with Ollama or LlamaEdge and its API server.
3. On a Mac with Apple Silicon, consider Rapid-MLX alongside the general tools.
4. For a ready-made bundle of chat UI, search and voice, try Harbor, if you are comfortable with Docker.
5. Use the table below to compare pricing and limits.

Check it worked by choosing one runner and noting why.

Step 3: Install the runner

Install only one runner at first, so you can tell what is causing any problem.

1. Download the app or install the command-line tool from the vendor's site, following its own instructions.
2. Open it and confirm it starts. Ollama and LM Studio both have desktop apps, and Rapid-MLX can be installed with a one-line script, Homebrew or pip.
3. If you use LlamaEdge, expect command-line setup: you run its API server on the WasmEdge runtime with a GGUF model file.
4. If you use Harbor, make sure Docker is running first.

Check it worked by running the tool's version or help command, or by seeing its window open.

Step 4: Download a model

Choose an open model that fits your memory and your job.

1. Browse the tool's model library or catalog. Ollama's listing names families such as Gemma, DeepSeek and Qwen, and Rapid-MLX has a catalog of MLX-native models.
2. Prefer a smaller or more compressed version if your memory is tight.
3. Read the model's licence, especially if you plan commercial use.
4. Start the download and wait. Large files can take a while on a slow connection.

Check it worked by seeing the model listed as installed in the app or CLI.

Step 5: Chat with it and test it

Run real prompts before building on top.

1. Open a chat or run the model from the command line.
2. Try a few prompts from your actual job: a document summary, a code question, a classification task.
3. Watch the speed. Local speed depends on your hardware, so if responses are slow, try a smaller model.
4. Note any wrong or invented answers. Local models can be wrong too.

Check it worked by getting a sensible answer to three of your own prompts.

Step 6: Connect it to your code or tools

Turn the chat into a service your other software can use.

1. Start the local server. LM Studio can serve several models through an OpenAI-compatible API, and Ollama exposes a REST API.
2. In your code or tool, set the base URL to the local address and use any key the tool asks for.
3. Run a simple request and confirm a response.
4. For coding agents, check the runner's documented launchers. Ollama, Harbor and Rapid-MLX each list a launch command for coding tools.
5. If you need documents, retrieval or tools, put PrivateGPT in front of your local server.

Check it worked by calling the endpoint from a script and printing the reply.

Step 7: Keep it private and maintain it

Local does not mean automatically safe, so check a few settings.

1. Keep the server bound to your own machine unless you mean to share it. Harbor warns that its built-in tunnel exposes your machine to the internet and needs authentication.
2. Review telemetry settings. Rapid-MLX's listing says its desktop app sends anonymous usage stats by default, which you can turn off.
3. Update the runner and models from time to time.
4. Delete models you no longer use to free disk space.

Check it worked by confirming the server is not reachable from another device unless you intended it.

Tools for this job compared

Tool	Good fit for	Pricing	Stated limits
Ollama	CLI, desktop app and REST API on macOS, Windows and Linux	Open source, paid from $20/mo (checked 24 Oct 2026)	Free accounts get starter cloud usage for a smaller set of models; local use on your hardware is unlimited
LM Studio	Desktop chat app with a local OpenAI-compatible server	Freemium, paid from $20/mo (checked 24 Oct 2026)	Local speed and model size depend on your hardware; Free plan has limited web search
Rapid-MLX	Apple Silicon Macs, OpenAI and Anthropic-compatible server	Open source	Needs an Apple Silicon Mac on macOS 14 or later; Intel Macs are not supported
Harbor	A prewired local stack of backends, chat UIs and tools	Open source	Requires Docker; running many backends at once is not practical on typical hardware
LlamaEdge	Small portable runtime for CPUs, GPUs and edge devices	Open source	Setup runs through the command line and WasmEdge; models must be in GGUF format
PrivateGPT	Files, retrieval and tools over a self-hosted local server	Open source	Does not run models; needs a separate OpenAI-compatible inference server

Prices and plan limits change, so confirm the current terms on each vendor's site before you commit.

Common mistakes to avoid

• Downloading a model too big for your machine. It loads slowly or not at all. Fix it by checking your memory against the model size first and starting small.
• Expecting cloud-model quality from a small local model. Smaller models make more mistakes. Fix it by testing on your own tasks and moving up a size only if you need to.
• Installing several runners at once. Each stores its own models and ports, which makes problems hard to trace. Fix it by using one runner until it works, then adding more deliberately.
• Exposing the local server to a network. An open port with no authentication lets others use your machine. Fix it by keeping it on localhost and adding authentication before any sharing.
• Skipping the licence. Open does not always mean free for every use. Fix it by reading the model's licence, and have a person check anything legal, medical or financial the model produces.

How we put this together

This is an evidence-based guide built from each tool's listing in our directory, which is compiled from the vendor's own site and pricing page. It is not a review based on using the tools. We don't rank tools, and nobody paid to be included. Some outbound links may be affiliate links; see our disclosure.

Frequently asked questions

Can I run an LLM locally for free?

Yes. Ollama says running models on your own hardware is unlimited, LM Studio's Free plan costs $0 and covers local models, and Rapid-MLX, Harbor, LlamaEdge and PrivateGPT are open source. You pay for the computer, the electricity, and any optional cloud models.

What hardware do I need to run an LLM locally?

It depends on the model size. LM Studio notes that local speed and model size depend on your own hardware, and Rapid-MLX requires an Apple Silicon Mac. Check the memory needs of your chosen model before downloading.

Is running an LLM locally private?

Local use keeps prompts on your device. LM Studio states data stays on your device when you use models locally. Cloud features and telemetry settings are separate, so check them in each tool.

Which tool is easiest for a beginner?

A desktop app is the easiest start, and LM Studio and Ollama both have one. Command-line tools and Docker-based stacks like Harbor suit people comfortable with a terminal.

Can I use a local model with my existing OpenAI-based code?

Often yes. Ollama, LM Studio, Rapid-MLX and LlamaEdge list OpenAI-compatible APIs, so you usually change the base URL. Test the features you rely on, because compatibility can be partial.

Compare more options in our Developer platforms category.