Local LLMs with Ollama
Ollama is the standard tool for running a large language model entirely on your own computer. Nothing you type leaves the machine, and install plus a first model run takes about 10-15 minutes.
Why Run Locally
- Privacy: input and output stay on your machine, which matters for confidential or personal material
- No usage billing: no per-token API charge — you pay only for electricity
- Offline: once a model is downloaded it runs without an internet connection
It does not replace everything. You are limited by your own hardware, and the best cloud models still lead on peak accuracy. Use local for light work, confidential data, and experiments; use the cloud when top accuracy matters.
Hardware Guide
| Model size | Rough memory and speed |
|---|---|
| 1B-4B (small) | About 8 GB. Runs without a GPU. Start here |
| 7B-8B (standard) | About 16 GB. Comfortable on Apple Silicon (M1 or later) or an RTX 3060 class GPU |
| 12B-14B (medium) | About 24 GB. M2 Pro or RTX 4070 class. Some waiting |
| 27B-32B (large) | 32 GB or more. M3 Max or RTX 4090 class. Heavy |
These are starting points — quantization, which Ollama applies by default, shifts the numbers. When in doubt, begin with a model around 4B and work upward.
Install
Ollama ships an official desktop app. On macOS (12 or later) and Windows, the installer at ollama.com is the easiest route, and Apple Silicon is recommended. On Linux, run the install script:
curl -fsSL https://ollama.com/install.sh | sh
Confirm the install with ollama --version. Homebrew also works on a Mac, but the official app handles GPU setup and auto-update for you.
First Model Run
Run a model with ollama run <model>. The first run downloads several GB; later runs start instantly and drop you into an interactive shell.
# Gemma 3 (Google — light, reads images, a good first try) ollama run gemma3:4b # Qwen 3 (Alibaba — strong general-purpose multilingual model) ollama run qwen3:8b # gpt-oss (OpenAI's open-weight model, strong at reasoning) ollama run gpt-oss:20b # DeepSeek-R1 (a reasoning model that thinks step by step) ollama run deepseek-r1:8b
The part after the colon is the size. Name it explicitly on a small machine, or you get the default size. Type /bye to leave the shell. The desktop app offers the same models in a chat window, with PDF and image drag-and-drop and a context-length slider.
Choosing a Model by Task
| Task | Suggested tags |
|---|---|
| General chat, Japanese | qwen3, gemma3 |
| Code generation | qwen3-coder |
| Reasoning and math | deepseek-r1, gpt-oss |
| Reading images | gemma3 (4B or larger) |
| Smallest and fastest | gemma3:1b, qwen3:0.6b |
These suggestions are limited to the current generation of tags — gemma3, qwen3, and gpt-oss. Tags move quickly, so check the official library at ollama.com/library when in doubt. Note that the small DeepSeek-R1 tags (7B, 8B) are distilled versions; the full R1 is a 671B model that will not run on a personal machine.
Calling It as an API
While Ollama runs, it exposes a REST endpoint at localhost:11434 that is OpenAI-compatible. Point existing code at it by changing one line:
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:11434/v1",
apiKey: "ollama", // any string works
});
const res = await client.chat.completions.create({
model: "qwen3:8b",
messages: [{ role: "user", content: "Three things to see in Tokyo?" }],
});
console.log(res.choices[0].message.content);
Ollama also supports structured output, which forces a fixed shape such as JSON, and tool calling, which lets the model trigger a search or a calculation. Both work locally, so an app can rely on a fixed data format without a cloud call.
Custom Models with a Modelfile
# save as Modelfile FROM qwen3:8b SYSTEM "You are a polite assistant who explains jargon in plain words." PARAMETER temperature 0.7 ollama create my-assistant -f Modelfile ollama run my-assistant
This fixes the starting behaviour — a role, a temperature — rather than changing what the model can do. Building one per task (translation, summarizing, code) saves retyping the same preamble every time.
GUI Tools
- Official Ollama app: chat UI, PDF and image drag-and-drop, context-length slider (Mac and Windows)
- Open WebUI: ChatGPT-like web UI with multi-user support and history management
- LM Studio: a separate GUI tool with visual model discovery and quantization choice
Ollama vs vLLM
| Ollama | vLLM |
|---|---|
| Personal, small-scale, experimental | Production services and high request volume |
| Set up in minutes | More setup work, including GPU configuration |
| Runs on consumer GPUs and Macs | Shows its value on enterprise GPUs |
Develop and validate on Ollama, then move to vLLM when concurrency grows. There is no need to start with vLLM.
Cautions
- Power and heat: a large model drives the GPU hard. Keep a laptop plugged in
- First download: models run from a few GB to tens of GB — leave room on disk
- Licensing: commercial use differs per model. Check each model's official license before using it at work
- Accuracy gap: local models still trail cloud flagships such as GPT-5.6 and Claude Opus 5 on the hardest work. Verify anything important
Next Step
Install the app, start with something small such as ollama run gemma3:4b, then pick models by task and wire one into your own code through the API.
