Skip to main content
The dispatch
Issue #11 2026-05-03 Sundays only

Reply Judge sees what humans miss.

A second-pass scoring model isn't redundant — it catches drift the planner can't. Three patterns the judge has surfaced in the last month that no operator caught first.

A second model that only checks the first one's work sounds redundant until you watch it catch things humans don't. This week: three brand-voice drifts Reply Judge surfaced last month that no operator had noticed.

Three catches

  • A creeping 'we sincerely apologize' formality in a brand whose anchor is warm and plain — invisible reply by reply, obvious in aggregate.
  • An over-eager discount offer pattern that drifted past policy during a busy week.
  • A subtle shift to passive voice in refund explanations that made the brand sound evasive.

None of these were wrong answers. They were right answers in a slightly wrong voice — exactly the failure mode a single helpfulness-optimized model can't see in itself. The judge isn't redundant; it's a different question asked by a different model.

Drift is invisible one reply at a time and obvious across two hundred. The judge reads the two hundred.

// three links we sent

  • An essay on evaluator modelsaround the web
  • A study on tone perception in supportaround the web
  • Anthropic's docs on model-graded evalsdocs.anthropic.com

// one ship

Reply Judge now reports the specific lines that lowered a score

Held replies arrive with the exact phrases that pulled fidelity down.

See the changelog

// the dispatch

Get this Sunday's issue.

Subscribe and the next dispatch lands at 09:00 local this Sunday. Operator essay, three ecosystem links, ship of the week. That's it.

Sundays only · One-click unsubscribe · No tracking pixels