Cases Where Fine-Tuning (FT) Works
FT is technology to "customize the LLM's behavior." Not everything is solved by FT; there are suited and unsuited cases.
Cases FT Suits
- Want to teach a company-specific tone
- Want a special output format kept every time
- Want to raise accuracy of fixed tasks like classification/extraction
- Want familiarity with a specific domain's jargon
- Want to shrink model size to lower inference cost
Cases FT Doesn't Suit
- Want to add new knowledge → RAG is more effective
- Frequently updated info → RAG (no retraining)
- One-time customization → prompt engineering suffices
Main Methods
Full Fine-tuning
Re-train all model parameters. Dozens of GPUs × days to weeks. Not realistic for frontier models. Barely possible for mid-size models like Llama 3 8B.
LoRA (Low-Rank Adaptation)
Freeze the model body, train only added small matrices. Possible on 1-2 GPUs × a few hours. The most practical.
- Storage: tens of MB (full model is tens of GB)
- Training speed: 3-10x faster
- Easy switching: select among multiple LoRAs at load
QLoRA
LoRA + 4-bit quantization. Further reduces GPU memory, enabling FT of a 70B model even on a consumer GPU (RTX 4090).
Adapter / Prefix Tuning
A lightweight method close to LoRA. LoRA / QLoRA are more mainstream now.
How to Make Training Data
Format
JSONL (1 sample per line) is standard:
{"messages": [
{"role": "system", "content": "You are a polite sales rep"},
{"role": "user", "content": "Please give a quote"},
{"role": "assistant", "content": "Certainly. ..."}
]}Count Guide
- Tone: 100-500
- Classification task: 1,000-5,000
- Extraction task: 500-2,000
- Specialized domain: 3,000-10,000
Quality vs Quantity
Quality matters more than count. If 10% of 1,000 items have errors, the model learns those errors.
Main Providers
| Provider | Supported models |
|---|---|
| OpenAI Fine-tuning | GPT-4o, GPT-4o Mini |
| Anthropic FT | Limited, Claude Haiku |
| Google Vertex AI | Gemini Flash |
| Together AI | Llama, Mistral, Qwen |
| Fireworks AI | Many open models |
| OpenPipe | Distillation + FT workflow |
| Self-host | Hugging Face PEFT, Axolotl, Unsloth |
Cost
- OpenAI gpt-4o-mini FT: $3.75/1M training tokens
- Together AI Llama 3 8B: ~$1.5/1M tokens
- Self-host QLoRA: a few GPU hours rental ($5-10 on H100)
FT Procedure
- Narrow the use (tone? classification? format?)
- Prepare training data (start with 100-1000)
- Carve out 10-20% validation data
- Run FT, monitor the loss curve (mind overfitting)
- Evaluate with the Golden Set, confirm improvement
- Staged release to production, A/B test
- Prepare a rollback procedure
Failure Patterns
- Only increasing count without looking at quality
- Overfitting into "answers only in the training-data tone"
- Thought you taught new knowledge, but it's old at inference
- FT results obsolete when the model generation changes
Summary
FT shows power in fixing style/tone/output format, but knowledge addition prioritizes RAG. LoRA / QLoRA make it practical even on personal GPUs. Starting from 100-1000 high-quality data items is the realistic answer.
