Project: eval-harness · Branch: feat/parallel-runner · Mode: default · Model: qwen3-coder

Summary

Rewrote the runner to evaluate cases concurrently with asyncio and a bounded semaphore, exponential backoff with jitter on 429/5xx, and resumable runs from a JSONL checkpoint. A 2,000-case suite went from 46 to 7 minutes.

Conversation

Turn 1 — User

The eval suite takes 46 minutes. Make it parallel and resumable.

Turn 1 — Assistant

Async runner with a semaphore (default 16), retries with full jitter on 429 and 5xx only, and a JSONL checkpoint so a crashed run resumes from the last finished case.

Tools used in this turn:

- Edit: harness/runner.py
- Write: harness/checkpoint.py
- Bash: python -m harness run suites/support.yaml — 2000 cases in 7m02s

Connections

- eval-harness
- Asyncio
- Backoff
- Checkpointing