sessionwiki

Session: run-evals-in-ci-on-prompt-changes — 2026-11-04

eval-harness · Claude · branch chore/ci · claude-sonnet-4-6 · 2026-11-04 10:40 · 1 msgs · 3 tools · 1 min read · updated yesterday

tools: Bash, Edit, Write
← eval-harness Download .md .txt .json
Tool calls
Bash Bash: 1 call (33.3%) 1 Edit Edit: 1 call (33.3%) 1 Write Write: 1 call (33.3%) 1
Token usage72.8K total
Input7.8K
Cache creation15.6K
Cache read46.8K
Output2.6K
Cache hit ratio67%· warming up

Project: eval-harness · Branch: chore/ci · Mode: default · Model: claude-sonnet-4-6

Summary

Wired the harness into CI: a workflow runs the smoke suite on every PR that touches prompts/, posts the score deltas as a PR comment, and fails if faithfulness drops more than two points.

Conversation

Turn 1 — User

Run evals automatically when someone changes a prompt.

Turn 1 — Assistant

Path-filtered workflow on prompts/**, smoke suite (200 cases), baseline from the latest main run, comment with the delta table, and a threshold gate on faithfulness.

Tools used in this turn:

  • Write: .github/workflows/evals.yml
  • Edit: harness/cli.py — --fail-on-drop
  • Bash: act pull_request -W .github/workflows/evals.yml — ok

Connections

  • [[eval-harness]]
  • [[GitHubActions]]
  • [[QualityGates]]

Keyboard shortcuts

⌘K / Ctrl+KOpen command palette
/Focus search
g hGo to home
g pGo to projects
g sGo to sessions
j / kNext / prev row (tables)
?Show this help
EscClose dialogs

Structured queries

Mix key:value filters with free text in the palette:

type:sessionOnly session pages
project:llm-wikiFilter by project name (substring)
model:claudeFilter by model name (substring)
date:>2026-03-01Sessions after a date
date:<2026-04-01Sessions before a date
tags:rustPages mentioning a tag/topic
sort:dateSort results by date (newest first)

Example: type:session project:llm-wiki date:>2026-04 sort:date