Project: eval-harness · Branch: feat/faithfulness · Mode: default · Model: claude-opus-4-7

Summary

Added a faithfulness metric for RAG answers: claims are extracted from the answer, each claim is checked against the retrieved passages by an LLM judge with a fixed rubric, and the score is the supported-claim ratio with per-claim evidence in the report.

Conversation

Turn 1 — User

I want a faithfulness score for our RAG answers, with evidence I can inspect.

Turn 1 — Assistant

Two stages: claim extraction, then claim-vs-context judgment with a strict JSON rubric. Cache judge calls by hash so re-runs are free. Report keeps the supporting passage per claim.

def faithfulness(answer: str, contexts: list[str], judge: Judge) -> MetricResult:
    claims = extract_claims(answer)
    verdicts = [judge.supported(claim, contexts) for claim in claims]
    supported = sum(v.supported for v in verdicts)
    return MetricResult(
        name="faithfulness",
        score=supported / len(claims) if claims else 1.0,
        details=[v.as_dict() for v in verdicts],
    )

Tools used in this turn:

- Write: harness/metrics/faithfulness.py
- Write: harness/judge/rubrics/faithfulness.md
- Bash: pytest tests/metrics -q — 14 passed

Connections

- eval-harness
- RAG
- LLMAsJudge
- Faithfulness