How to Run an LLM Locally: Install, Load a Model, and Chat
You can run an LLM locally by installing a local inference runtime (Ollama is the simplest starting point), pulling a quantized model that fits your available VRAM or RAM, verifying the model responds from the command line, and optionally attaching a web interface like Open WebUI for a chat-style experience. This works on a modern laptop or desktop with enough memory; the main constraint is how much VRAM (GPU) or system RAM you have, since that determines the largest model you can load.
Step 1: Check your hardware before downloading anything
Model size is the deciding factor. A model that doesn't fit in memory will either fail to load or fall back to slow CPU inference.
| Your available memory | Practical model range | Notes |
|---|---|---|
| 8 GB VRAM | 7B–8B quantized | Comfortable for most 7B/8B models at 4-bit quantization |
| 12–16 GB VRAM | up to ~13B quantized | Good balance of quality and speed |
| 24 GB VRAM | up to ~30B quantized | Larger models become usable |
| CPU-only, 16 GB RAM | 7B quantized | Works, but expect slow token generation |
Quantization (often labeled Q4, Q5, Q8) shrinks a model's memory footprint at some cost to quality. A 4-bit (Q4) version of a 7B model is the common entry point because it fits in modest hardware while staying usable.
Check your GPU memory first (on NVIDIA, nvidia-smi; on macOS, the unified memory reported in System Information). If you have no discrete GPU, plan around a 7B quantized model and accept slower output.
Step 2: Install a local inference runtime
Ollama is the lowest-friction option and is the runtime most commonly paired with Open WebUI. Install it from the official Ollama site for your OS, then confirm the service is running:
ollama --version
Expected result: the version prints without error. If the command isn't found, the install didn't complete or the binary isn't on your PATH.
Alternatives exist if you want more control over the inference stack — llama.cpp is a common choice for running GGUF-format models directly — but Ollama handles model download, quantization selection, and serving for you, which is why it's the recommended starting point.
Step 3: Pull a model and confirm it loads
Download a quantized model by name. For example, a 7B-class model:
ollama pull <model-name>
The pull downloads the model file to disk. Two things to watch:
- Disk space: quantized 7B models are typically several GB; larger models scale up from there. Confirm free space before pulling.
- Memory at load time: the download succeeding does not mean the model will run. Loading is where VRAM/RAM limits surface.
After the pull completes, list what you have:
ollama list
Expected result: your model appears in the list with its size.
Step 4: Run your first inference from the command line
Start an interactive session:
ollama run <model-name>
Then type a prompt and press Enter. Expected result: the model streams a response back in the terminal.
If it works, you've confirmed the full path — runtime, model file, and inference — before adding any UI layer. This isolates problems: if something breaks later, you know the core stack is fine.
Common failures and what they mean
- Model loads but generates very slowly: it's likely running on CPU instead of GPU. Check that your GPU is detected by the runtime.
- Out-of-memory error on load: the model is too large for your VRAM/RAM. Drop to a smaller parameter count or a more aggressive quantization (e.g., Q4 instead of Q8).
- Pull stalls or fails: usually a network or disk-space issue, not a model problem.
- Command not found: the runtime isn't installed or isn't on your PATH.
Step 5: Add a web interface (optional)
The command line is enough to verify everything works, but a browser-based chat interface is more comfortable for ongoing use. Open WebUI is a self-hosted option that connects to a local runtime and provides a chat experience, prompt management, and model switching.
Open WebUI's own description frames it as running AI "on your own terms," connecting any model and extending with code — which matches the local-first workflow here: your runtime serves the model, and the interface sits on top of it.
The general pattern is: keep your inference runtime running, then point the web interface at it. Once connected, you select your pulled model from a dropdown and chat in the browser instead of the terminal.
What to decide before you start
- If you have a GPU with 8 GB+ VRAM: start with a 7B–8B quantized model via Ollama, verify with
ollama run, then add Open WebUI. - If you're CPU-only: expect slower responses; a 7B quantized model is still the realistic starting point.
- If you want maximum control over quantization and formats: consider llama.cpp instead of Ollama, at the cost of more manual setup.
- If you only need to test whether local inference works at all: stop after Step 4 — the command line already proves the stack.
The order matters: hardware check → runtime install → model pull → command-line verification → optional UI. Skipping the command-line step makes it much harder to tell whether a later problem is the model, the runtime, or the interface.