Local LLMs with Ollama
Ollama is the standard tool to run LLMs locally. With 5-minute install and 10-minute first model run, you get "AI that runs on your own PC."
Why Run Locally
- Privacy: data doesn't leave (important for medical/legal)
- Cost: no API fees. Just electricity
- Offline: no internet needed
- Customization: freely switch/fine-tune models
- No rate limit: call as much as you want
Required Specs
| Model size | RAM | GPU (recommended) | Speed feel |
|---|---|---|---|
| 3B | 8 GB | Not needed | Quite fast |
| 7B | 16 GB | M1 / RTX 3060 | Normal |
| 13B | 24 GB | M2 Pro / RTX 4070 | Slowish |
| 70B | 64+ GB | M3 Max / RTX 4090+ | Quite slow |
Install
Mac / Linux
brew install ollama # or curl -fsSL https://ollama.com/install.sh | sh
Windows
Download the installer from ollama.com
First Model Run
# Llama 3.3 (Meta, general) ollama run llama3.3 # Qwen 2.5 (Alibaba, multilingual) ollama run qwen2.5 # Gemma 3 (Google) ollama run gemma3:7b # DeepSeek-R1 (reasoning model) ollama run deepseek-r1
The first time is a GB-scale download. From the 2nd time it launches instantly. An interactive shell starts and you can converse on the spot.
Use as an API
Launching Ollama starts a REST API at localhost:11434. Callable as an OpenAI-compatible API:
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:11434/v1",
apiKey: "ollama", // any string
});
const res = await client.chat.completions.create({
model: "llama3.3",
messages: [{ role: "user", content: "Things to see in Tokyo?" }],
});
How to Choose Main Models
| Use | Recommended |
|---|---|
| General chat | llama3.3, qwen2.5 |
| Japanese | qwen2.5, llm-jp-3 |
| Code generation | qwen2.5-coder, deepseek-coder |
| Reasoning/math | deepseek-r1 |
| Image understanding | llava, qwen2.5-vl |
| Small/fast | llama3.2:1b, gemma3:1b |
GUI Tools
- Open WebUI: use Ollama with a ChatGPT-like UI
- LM Studio: Ollama alternative, rich GUI
- Msty: GUI with easy multi-model switching
Difference from vLLM
| Ollama | vLLM | |
|---|---|---|
| Use | Personal/small | Production/high-volume |
| Setup | 5 min | 1 hour+ |
| Performance | Normal | Best |
| Required GPU | Consumer OK | Enterprise GPU recommended |
Customization
Custom Model with a Modelfile
# Modelfile FROM llama3.3 SYSTEM "You are a polite Japanese assistant." PARAMETER temperature 0.7 # Build ollama create my-assistant -f Modelfile ollama run my-assistant
Cautions
- Power consumption: 200-400W at full GPU load
- First DL: a few to tens of GB, mind line speed
- Commercial use: confirm license per model (Llama 3.3 is commercial-OK)
- Performance gap: doesn't reach API flagships (GPT-5, Claude Opus)
Next Step
Once used to local dev, production operation is in the build/ops chapter "Local LLM Operation."
