The LLM Rolls Dice
An LLM may return different output every time for the same input. This is because the LLM, after computing "the probability distribution of the next word," samples (draws lots) from that distribution. Parameters like temperature, top-p, top-k control how probability is rolled.
Temperature
A value 0 to 2. Controls how much to "sharpen / flatten" the probability distribution.
Example: The Word After "Japan's capital is"
| Candidate | Raw prob | After T=0.5 | After T=2.0 |
|---|---|---|---|
| Tokyo | 0.95 | 0.99 | 0.7 |
| Kyoto | 0.03 | 0.005 | 0.15 |
| Osaka | 0.01 | 0.001 | 0.10 |
| Other | 0.01 | 0.001 | 0.05 |
Temperature Guide
- 0.0: fully deterministic (same every time). For tests/classification.
- 0.0-0.3: nearly deterministic. Fact-check, extraction, summary.
- 0.5-0.7: standard. Chat, Q&A, coding.
- 0.8-1.0: creative. Novels, poetry, brainstorm, ad copy.
- 1.0+: very diverse, quality-degradation risk.
top-p (Nucleus Sampling)
Collect top words until their probabilities sum to p, then choose within that range. 0.0-1.0.
- 0.1: only the very top, nearly deterministic
- 0.5: moderate
- 0.9: standard (many models' default)
- 1.0: choose from all candidates
Temperature and top-p can be used together, but it's common to move only one. OpenAI officially recommends "don't change both."
top-k
Choose only from the top k words by probability.
- k=1: always the max-likelihood candidate (greedy)
- k=10-50: standard
- k=100+: diversity-focused
Compared to top-p, being a fixed number, it's harder to adapt to distribution sharpness by context. top-p is more mainstream now.
Other Parameters
frequency_penalty / presence_penalty
Suppress repetition of the same word. 0.0-2.0. Try around 0.5 when repetition stands out in text generation.
max_tokens
Max output tokens. For billing limits and preventing cut-offs.
stop sequence
Stop generation when a specific string like "###" appears. Useful for format control.
Recommended Settings by Use
| Use | temperature | top-p |
|---|---|---|
| Classification/extraction | 0.0 | 1.0 |
| Fact-check Q&A | 0.1-0.3 | 1.0 |
| Code generation | 0.2-0.5 | 0.95 |
| Translation/summary | 0.3-0.5 | 1.0 |
| Customer support | 0.5-0.7 | 0.9 |
| Marketing copy | 0.7-0.9 | 0.9 |
| Creative/story | 0.8-1.0 | 0.95 |
| Brainstorm | 0.9-1.0 | 1.0 |
Probability-Control Pitfalls
- Some models aren't fully deterministic even at temperature 0 (implementation-dependent)
- Same parameters behave differently across models
- Using without checking the API default causes variance
- In production, fixing the seed raises reproducibility (OpenAI etc.)
2026 Tendencies
- Reasoning models (thinking line) do multi-stage sampling internally
- Many frontier models default to "auto-adjustment"
- Training to keep consistency even above temperature 0 advances
Summary
Temperature/top-p are parameters controlling the LLM's "way of rolling dice." By use, classification at low temperature, creative at high is basic. Don't change both at once; fixing one and adjusting the other is recommended.
