judge-integrity

judge-integrity

A judge is a measurement instrument. An instrument that returns zero when it is broken is worse than no instrument.

Contract

• Every judge attempt returns a status: ok, unparseable, call_failed, or timeout.
• scores is present only when status is ok. A parse failure is never a zero.
• Transport errors retry with backoff. Refusals and parse failures do not retry into another score.
• A deterministic verifier runs before the paid judge. A definitive pass may skip the judge; unknown falls through.
• Judge and evaluation calls pin temperature 0 and preserve structural markers.
• Escalate to a stronger judge only below an explicit score threshold.
• Store the judge prompt hash and status beside the score.

Procedure

1. Parse the response with balanced-brace extraction; fences and nested objects are normal.
2. Run the task verifier. Stop on pass or fail only if the task declares the verifier authoritative.
3. Call the judge with a timeout. Record status and duration even on failure.
4. Store scores, status, model, and prompt hash. Never overwrite the generation record.
5. Re-score stored output with a separate command; do not re-run generation to re-judge.

Worked example

Three stored judge records: scored, unparseable, and call_failed.

{"status": "scored", "score": 7.5, "model": "judge-v2", "prompt_hash": "sha256:..."}
{"status": "unparseable", "score": null, "model": "judge-v2"}
{"status": "call_failed", "score": null, "error": "timeout"}

Anti-patterns

• Mapping unparseable output to 0.0 and calling it a score.
• Treating a provider outage as a low-quality answer.
• Re-running the full generation to apply a new judge.
• Pooling judge scores across model revisions or rubric versions.

References

• status

Verification Checklist

• [ ] A raising provider yields call_failed, not a score. (step 3)
• [ ] Prose output yields unparseable, not a score. (step 1)
• [ ] Nested JSON parses correctly. (step 1)
• [ ] Re-scoring a stored execution performs no completion call. (step 5)
• [ ] Every score has a status and a model fingerprint. (step 4)

Composes With

• structured-output-contract — parsing and constraint precedence
• execution-observability — judge status and request metadata
• feedback-loop — calibration against human feedback