Project: eval-harness · Branch: chore/ci · Mode: default · Model: claude-sonnet-4-6

Summary

Wired the harness into CI: a workflow runs the smoke suite on every PR that touches prompts/, posts the score deltas as a PR comment, and fails if faithfulness drops more than two points.

Conversation

Turn 1 — User

Run evals automatically when someone changes a prompt.

Turn 1 — Assistant

Path-filtered workflow on prompts/, smoke suite (200 cases), baseline from the latest main run, comment with the delta table, and a threshold gate on faithfulness.

Tools used in this turn:

- Write: .github/workflows/evals.yml
- Edit: harness/cli.py — --fail-on-drop
- Bash: act pull_request -W .github/workflows/evals.yml — ok

Connections

- eval-harness
- GitHubActions
- QualityGates