Prompt Evaluation Basics: Reproducibility and Accuracy
"Somehow it got better" doesn't fly in development. Prompt changes need their quality measured by evaluation.
Building the Evaluation
- Evaluation dataset: collect representative inputs and expected outputs
- Metrics: accuracy, format compliance, refusal rate, cost, latency
- Automatic scoring: exact-match/rule/another-LLM scoring (LLM-as-judge)
- Regression tests: re-run on prompt/model changes
Caution
- Don't judge from a few impressions. Compare with data
- LLM-as-judge errs too; for key metrics also use human sample checks
- Performance changes with model updates; run evaluation continuously
Start Minimal
A perfect evaluation platform is unnecessary. First make 20–50 representative cases + expected values and just run them before/after changes—incidents drop sharply.
Key Point
Prompts are "code." Don't deploy without tests. The next chapter covers evaluation and management tools.
