Prompt Management: Promptfoo / LangSmith / Helicone
To run prompts in production, you need tools for evaluation, observability, and version management. Here's the role of representative examples.
Difference in Roles
- Promptfoo: comparative evaluation/regression testing of prompts/models (CI-embeddable)
- LangSmith: tracing, debugging, evaluation (execution visualization)
- Helicone: observability of API calls, cost/latency measurement, caching
Order of Adoption
- First visualize observability (what's called, how much it costs)
- Next evaluation (a base to measure if changes are good/bad)
- Then version management (manage prompts as assets)
Caution
- When sending confidential prompts/data to a tool, confirm its data-handling terms
- Features/pricing change; confirm the latest officially
- Tools are means; evaluation design (metrics, data) is the substance
Chapter Summary
Prompts are "code that is tested, observed, and version-managed." Improvement without measurement isn't improvement—this is the conclusion of the prompt-design chapter.
