{
  "id": "eval-harness/rag-faithfulness-metric",
  "slug": "rag-faithfulness-metric",
  "title": "Session: rag-faithfulness-metric — 2026-09-04",
  "type": "source",
  "project": "eval-harness",
  "date": "2026-09-04",
  "started": "2026-09-04T16:00:00+00:00",
  "ended": "2026-09-04T18:36:00+00:00",
  "model": "claude-opus-4-7",
  "cwd": "/home/dev/code/eval-harness",
  "git_branch": "feat/faithfulness",
  "permission_mode": "default",
  "user_messages": 1,
  "tool_calls": 3,
  "tools_used": [
    "Bash",
    "Write"
  ],
  "is_subagent": false,
  "wikilinks_out": [
    "Faithfulness",
    "LLMAsJudge",
    "RAG",
    "eval-harness"
  ],
  "body_text": "Project: eval-harness · Branch: feat/faithfulness · Mode: default · Model: claude-opus-4-7\n\nSummary\n\nAdded a faithfulness metric for RAG answers: claims are extracted from the answer, each claim is checked against the retrieved passages by an LLM judge with a fixed rubric, and the score is the supported-claim ratio with per-claim evidence in the report.\n\nConversation\n\nTurn 1 — User\n\nI want a faithfulness score for our RAG answers, with evidence I can inspect.\n\nTurn 1 — Assistant\n\nTwo stages: claim extraction, then claim-vs-context judgment with a strict JSON rubric. Cache judge calls by hash so re-runs are free. Report keeps the supporting passage per claim.\n\ndef faithfulness(answer: str, contexts: list[str], judge: Judge) -> MetricResult:\n    claims = extract_claims(answer)\n    verdicts = [judge.supported(claim, contexts) for claim in claims]\n    supported = sum(v.supported for v in verdicts)\n    return MetricResult(\n        name=\"faithfulness\",\n        score=supported / len(claims) if claims else 1.0,\n        details=[v.as_dict() for v in verdicts],\n    )\n\nTools used in this turn:\n\n- Write: harness/metrics/faithfulness.py\n- Write: harness/judge/rubrics/faithfulness.md\n- Bash: pytest tests/metrics -q — 14 passed\n\nConnections\n\n- eval-harness\n- RAG\n- LLMAsJudge\n- Faithfulness",
  "sha256": "62272c1158f5036f",
  "source_url": "sessions/eval-harness/2026-09-04-rag-faithfulness-metric.html"
}