Measure · Intermediate
Evaluation and reliability
Benchmarks, task-specific evals, LLM-as-judge and red-teaming, plus why leaderboards mislead.
Why it matters
You cannot improve what you cannot measure, and many published scores do not measure what they claim. Evals are the core skill of AI engineering.
Key ideas
- Benchmark design and contamination
- Task-specific eval sets
- LLM-as-judge and its biases
- Red-teaming
- Regression testing for prompts
You will be able to
- Build an eval set for your app
- Spot a contaminated benchmark
- Track quality across releases
Primary reading
- Holistic Evaluation of Language Models (HELM) (opens in a new tab)Liang et al. · 2022
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (opens in a new tab)Jimenez et al. · 2023
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (opens in a new tab)Zheng et al. · 2023