Project: eval-harness · Branch: feat/cost · Mode: default · Model: qwen3-coder

Summary

Tracked token usage and cost per eval run and per metric, with prices loaded from a versioned table, and added a budget flag that stops a run before it exceeds a limit.

Conversation

Turn 1 — User

How much does one full eval run cost us?

Turn 1 — Assistant

Record usage per judge call, price it from prices.yaml (versioned by date), aggregate per metric, and add --budget that aborts cleanly with partial results.

Tools used in this turn:

- Edit: harness/runner.py
- Write: harness/cost.py
- Bash: python -m harness run suites/support.yaml --budget 5 — $3.12

Connections

- eval-harness
- CostTracking
- Budgets