Advanced
8.5h · 38 lessons
Evaluating LLM Applications
Offline evals, online experiments, and safety testing.
4.8rating5,410 enrolledLifetime access
Start free preview
About this course
Build the eval discipline that separates hobby projects from real products. Metrics, judges, drift, and regression testing.
What you'll learn
- Design evaluation sets that actually predict user experience
- Combine rule-based, model-graded and human evaluation
- Track quality over time with dashboards and alerts
- Catch regressions before they reach production
Requirements
- Have shipped or prototyped at least one LLM feature
- Comfort with Python notebooks
- Access to your app's real (or realistic) inputs
Curriculum
6 modules · 24 lessons
Module 1Why evals matter4 lessons
Module 1
Why evals matter
- 01.01The vibes-driven trap
- 01.02Signal vs noise
- 01.03What good looks like
- 01.04Setting up an eval harness
Module quiz · 3 questions
Module 2Building datasets4 lessons
Module 2
Building datasets
- 02.01Mining real traffic
- 02.02Synthetic augmentation
- 02.03Labeling with humans
- 02.04Versioning datasets
Module quiz · 3 questions
Module 3Grading techniques4 lessons
Module 3
Grading techniques
Module quiz · 3 questions
Module 4CI and dashboards4 lessons
Module 4
CI and dashboards
Module quiz · 3 questions
Module 5Advanced topics4 lessons
Module 5
Advanced topics
Module quiz · 3 questions
Module 6Capstone4 lessons
Module 6
Capstone
Module quiz · 3 questions