What Is a Local LLM and How Do You Run One Yourself?

A local LLM is a large language model that runs on hardware you control—your laptop, desktop, or a server you manage—instead of being called through a cloud API. You run one by installing a model runner (Ollama and llama.cpp are the common choices), pulling a model file, and chatting with it through a terminal or a web interface like Open WebUI. The tradeoff is straightforward: you get privacy, offline operation, and no per-token billing, but you supply the RAM, VRAM, and patience. A local setup makes sense when your data shouldn't leave your machine, when you want to work without a connection, or when you're running enough volume that API costs would add up. It's the wrong choice when you need frontier-model reasoning quality or very large context windows.

What "local" actually changes

With a cloud model, your prompt travels to someone else's GPU and the response comes back over the network. With a local LLM, both the model weights and the computation live on your machine. That single difference drives everything else:

• Privacy — prompts and documents never leave your hardware. This is the main reason people choose local for sensitive material.
• Offline use — once the model is downloaded, no internet connection is required.
• No per-token cost — you pay in hardware and electricity rather than per request. There's no metered bill, but there's also no free lunch: the compute is yours to provide.
• Full control — you pick the model, the quantization, and the version, and nothing changes under you unless you change it.

The cost is capability. Small models that fit on consumer hardware are generally less capable than the largest hosted models, especially on hard reasoning, long documents, and complex multi-step tasks.

Hardware reality check

The single biggest constraint is memory. A model has to fit in RAM (or VRAM, if you're using a GPU) alongside the context you're generating. Rough guidance by model size, assuming 4-bit quantization, which is the usual default:

A few practical notes that the table hides:

• GPU vs CPU is a speed difference, not a capability difference. A 7B model on CPU will answer, just slowly—often a few tokens per second. The same model on a mid-range GPU can be several times faster.
• VRAM is tighter than system RAM. If a model doesn't fit in VRAM, the runner offloads layers to CPU and generation slows sharply. Partial offload works; it just costs speed.
• Context eats memory too. A long conversation or a big pasted document adds to the footprint, so a model that "just fits" may fail once you give it a long prompt.
• Quantization is the lever. A 4-bit version of a model is roughly a quarter the size of the full-precision version with a modest quality cost, which is why it's the standard choice for local use.

A typical modern laptop with 16 GB of RAM runs 7–8B models acceptably on CPU and comfortably if it has a discrete GPU. That's the realistic entry point for most people.

The toolchain: runner plus interface

Local LLM setups have two layers, and keeping them separate makes everything easier to reason about.

The runner loads the model and does the inference. Ollama is the most approachable—it manages model downloads, serves an API, and handles GPU/CPU placement for you. llama.cpp is lower-level and more configurable, and many other tools are built on it.

The interface is what you actually type into. The runner's own CLI works, but a web UI gives you chat history, multiple conversations, and model switching. Open WebUI is a self-hosted interface in this category: it connects to a runner like Ollama and gives you a browser-based chat experience on your own machine. The two pieces are independent—you can swap the interface without touching your models, or change runners without losing your chat setup.

Step-by-step first run

This path assumes a machine with enough memory for a small model (see the table above) and a terminal.

1. Install a runner. Install Ollama for your platform and confirm it's running. The install typically starts a background service that listens on a local port.

2. Pull a small model. Start with something in the 3–8B range so the first run succeeds quickly. Pulling downloads the model file; expect a few gigabytes.

3. Chat in the terminal. Send a prompt and watch the response stream. This verifies the model loads and generates before you add any other moving parts. If this step is slow but working, that's expected on CPU.

4. Add a web interface (optional). Once terminal chat works, install Open WebUI and point it at your runner. You get a browser chat UI with conversation history and model selection. Do this after the runner works, so if something breaks you know which layer is at fault.

5. Verify end to end. Ask a question you know the answer to, then ask a follow-up that depends on the first answer. If the follow-up works, context handling is fine. If it doesn't, you likely have a context or memory limit issue.

Common failure points and fixes

• Out-of-memory errors on load. The model is bigger than your available memory. Drop to a smaller model or a more aggressive quantization, and close other memory-hungry apps.
• It loads but generation is painfully slow. The model is running on CPU or partially offloaded. Check whether your GPU is being used; if not, the model may not fit in VRAM. A smaller model or lower quantization usually fixes this.
• Model won't load at all. Often a corrupted or incomplete download. Re-pull the model.
• Web UI can't reach the runner. Usually a port or host mismatch—the interface is looking at the wrong address, or the runner isn't running. Confirm the runner is up first, then check the address the UI is configured with.
• Works for short prompts, fails on long ones. Context is exceeding memory. Shorten the input or reduce the context window setting.

When a local LLM is the wrong choice

Be honest about the ceiling. Local models are a good fit for drafting, summarizing, classification, extraction, and privacy-sensitive work where "good enough" quality is acceptable. They're a poor fit when:

• You need frontier-model reasoning for hard, multi-step problems where small models visibly fall short.
• You need very large context—book-length documents or long codebases—since context is bounded by your memory.
• You need high throughput across many concurrent users, where a single local machine becomes the bottleneck.
• You want zero setup and maintenance, since you're now responsible for installs, updates, and troubleshooting.

A common pattern is hybrid: keep a local model for private or high-volume routine work, and reach for a hosted model when a task genuinely needs more capability. The two aren't mutually exclusive, and the runner-plus-interface split makes it easy to switch between them.