sessionwiki

Session: parallel-runner-with-retries — 2026-10-02

eval-harness · ZCode · branch feat/parallel-runner · qwen3-coder · 2026-10-02 09:20 · 1 msgs · 3 tools · 1 min read · updated 4 weeks ago

tools: Bash, Edit, Write
← eval-harness Download .md .txt .json
Tool calls
Bash Bash: 1 call (33.3%) 1 Edit Edit: 1 call (33.3%) 1 Write Write: 1 call (33.3%) 1
Token usage106.4K total
Input11.4K
Cache creation22.8K
Cache read68.4K
Output3.8K
Cache hit ratio67%· warming up

Project: eval-harness · Branch: feat/parallel-runner · Mode: default · Model: qwen3-coder

Summary

Rewrote the runner to evaluate cases concurrently with asyncio and a bounded semaphore, exponential backoff with jitter on 429/5xx, and resumable runs from a JSONL checkpoint. A 2,000-case suite went from 46 to 7 minutes.

Conversation

Turn 1 — User

The eval suite takes 46 minutes. Make it parallel and resumable.

Turn 1 — Assistant

Async runner with a semaphore (default 16), retries with full jitter on 429 and 5xx only, and a JSONL checkpoint so a crashed run resumes from the last finished case.

Tools used in this turn:

  • Edit: harness/runner.py
  • Write: harness/checkpoint.py
  • Bash: python -m harness run suites/support.yaml — 2000 cases in 7m02s

Connections

  • [[eval-harness]]
  • [[Asyncio]]
  • [[Backoff]]
  • [[Checkpointing]]

Keyboard shortcuts

⌘K / Ctrl+KOpen command palette
/Focus search
g hGo to home
g pGo to projects
g sGo to sessions
j / kNext / prev row (tables)
?Show this help
EscClose dialogs

Structured queries

Mix key:value filters with free text in the palette:

type:sessionOnly session pages
project:llm-wikiFilter by project name (substring)
model:claudeFilter by model name (substring)
date:>2026-03-01Sessions after a date
date:<2026-04-01Sessions before a date
tags:rustPages mentioning a tag/topic
sort:dateSort results by date (newest first)

Example: type:session project:llm-wiki date:>2026-04 sort:date