ExperimentPrototype
Golden-Set Evaluation Harness for Agents
Regression testing for prompts — a graded golden set built from real production queries, with per-category scoring so an aggregate average can't hide a broken edge case.
What I'm testing
Whether an LLM-as-judge grader stays correlated with human labels over time, and how often it needs recalibrating.
Finding so far
Rubric-based grading with explicit deterministic checks alongside (did it cite, did it stay in schema) is substantially more stable than asking for a holistic score.
Open question
How large the golden set needs to be before per-category scores stop being noisy. Currently at 150; categories with fewer than ~20 examples swing too much to gate on.
Stack
- Python
- BigQuery
- Cloud Run