Rate Limiting, Failover, and Redundancy
Assume LLMs are externally dependent, usage-billed, and occasionally down. In production, defensive design protects quality and cost.
Rate Limiting
- Control it yourself before hitting the provider's limit (queue/backoff)
- Per-user quotas to prevent abuse and cost runaway
Failover and Redundancy
- Switch to an alternate model during provider/model outages
- Timeouts and retries (exponential backoff; mind idempotency)
- Fallback on degradation (simple answer / route to a human)
Defending Cost
- Cap alerts; cut off unexpected spikes
- Use caching to avoid re-billing identical requests
Caution
Make changes with large production impact (model switch, cap change) presuming staged rollout and rollback. They only work when paired with observability (previous chapter).
Chapter Summary
The core of productionization is observability + defensive design. "It ran" and "it keeps running and doesn't break when it falls" are different things.
