Why Cost Optimization Matters
An LLM product's production cost becomes huge cumulatively. Even a few yen per request becomes millions to hundreds of millions of yen/month with users × frequency × 365 days. Being conscious of optimization can be 5-10x more efficient.
1. Prompt Caching
A mechanism cutting input-token cost up to 90% when reusing the same system prompt.
- Anthropic: explicit via cache_control parameter
- OpenAI: automatic caching (when the same prefix hits a certain number of times)
- Google: Context Caching
The effect is enormous in cases like agent operation or RAG where the system prompt and tool definitions are long.
⚠️ Cache-invalidation pitfall: mixing dynamic elements like datetime, username, session ID at the prompt head causes cache misses, applying normal new-token pricing. The fix is to design "static prefix → dynamic tail." In ProjectDiscovery's actual case, just moving working memory to the end of the message cut LLM cost by 59%.
2. Model Cascade
Selectively use multiple models in one app:
- Light (Mini, Nano, Haiku): classification, routing, simple extraction
- Mid (Sonnet, Mistral Large, Gemini Flash): daily work, summary, translation
- Frontier (GPT-5, Claude Opus 4.7): complex reasoning, code gen, agent commander
A config handling 80% with Light, 15% Mid, 5% Frontier cuts cost to 1/5-1/10 vs frontier-only.
Cascade Implementation
- Judge "is this a complex question" with a lightweight model
- Simple → complete with the lightweight model
- Complex → escalate to Mid, and to Frontier if still hard
- Log results and cost, train the judge model to improve
3. Batch API
Provided by OpenAI, Anthropic, Google. Processes non-real-time tasks 50% cheaper.
- Bulk translation, data classification, summarization
- Results return within 24 hours
- Ideal for nightly batch processing
4. Context Compression
- Pass summarized conversation history (summary mode over 10 turns)
- Pass only relevant parts via RAG (passing full text is wasteful)
- YAML/Markdown lighter than JSON in some cases
- Remove unneeded newlines/whitespace
5. Reranker for Accuracy + Cost Both
Retrieve 100 in RAG → narrow to top 5 with a Reranker (Cohere Rerank, Voyage Rerank) → pass to the LLM. Fewer passed tokens while accuracy rises.
6. Quantization (When Self-Hosting)
When self-hosting an open-weight model, quantization cuts memory and compute:
- int8: nearly maintains accuracy, half memory
- int4 (GPTQ, AWQ): 1/4 memory, 5-10% accuracy drop
- FP8: native support on H100/Blackwell, fast
vLLM, Together AI, Fireworks support quantization.
7. Distillation
Have a big model (GPT-5) generate lots of data, fine-tune a small model (Llama 8B) with it. Aim for big-model-level accuracy with a small model. OpenPipe, Together, Fireworks provide workflows.
8. Prompt Shortening
- "Please could you kindly..." → "..." (politeness unneeded)
- Narrow long examples to 1-2
- Chain-of-Thought unneeded for reasoning models (done automatically)
- Specifying output with JSON Schema stabilizes + shortens
9. Streaming Responses
Response time itself doesn't change, but UX improves → subscription-retention improves → ROI improves as a result.
10. Monitoring
After applying cuts, measure the effect:
- Average tokens per request
- Usage ratio by model
- Cache hit rate
- P95 latency
- Error rate (whether quality degraded from cost cuts)
Concrete Cut Examples
| Measure | Reduction |
|---|---|
| Prompt caching | 30-90% |
| Model cascade | 50-80% |
| Batch API | 50% |
| RAG + Reranker | 30-60% |
| Quantization (self-host) | 40-70% |
| Distillation (small model) | 80-95% |
Summary
Inference-cost optimization's big three are "caching + cascade + context compression." Realistically, apply gradually while watching monthly cost after going to production. Ultimately, systematizing optimization before cost reaches training-cost levels is important.
