Project: eval-harness · Branch: data/golden-set · Mode: default · Model: gemini-2.5-pro

Summary

Curated a 300-case golden dataset from support tickets: de-duplicated near-identical questions with embeddings, stratified by product area, and stored expected answers with source citations.

Conversation

Turn 1 — User

Our eval set is 80% password-reset questions.

Turn 1 — Assistant

Cluster ticket questions by embedding similarity, sample per product area, and keep the citation for each expected answer so faithfulness can be checked.

Tools used in this turn:

- Write: scripts/curate_golden.py
- Bash: python scripts/curate_golden.py --target 300 — suites/golden.yaml

Connections

- eval-harness
- Datasets
- Embeddings