BACKGROUND Large language models (LLMs) are increasingly used to evaluate AI-generated medical content, yet judge validity is commonly inferred from agreement with human ratings without fully characterizing the reliability of the human reference standard or the clinical direction of disagreement. OBJECTIVE This study aimed to evaluate LLM judges for surgical patient education using a validation framework that jointly considers human reference-standard reliability, judge-expert concordance, directional error, and safety-sensitive classification. We additionally examined whether these principles extend to a decision-native probabilistic evaluator. METHODS Five generative LLM judges evaluated 100 craniosynostosis patient-education responses generated by five contestant LLMs (20 questions × 5 generators); three neurosurgeons independently rated all responses on Accuracy, Safety, Empathy, and Completeness. Contestant responses were collected in January 2026, generative-judge evaluations in March 2026, and the Jev-1.13.0 benchmark in October 2026. We examined expert reliability, absolute agreement, directional error, cluster-bootstrap uncertainty, safety classification, self-preference, and a five-judge ensemble. The same benchmark was evaluated separately using Jev-1.13.0, a decision-native probabilistic evaluator that assigns probabilities to predefined ordinal scoring levels and binary clinical decisions, rather than generating single rubric scores and narrative rationales, to examine repeatability, typed safety decisions, calibration, and configuration sensitivity RESULTS Individual expert interrater reliability was limited (ICC[A,1]=0.322; three-expert mean ICC[A,3]=0.587). Generative judge–expert absolute-agreement ICCs ranged from 0.182 to 0.370. Opus-4.6 and GPT-5.4 had negative signed biases (−0.374 and −0.426); Llama-4 Maverick and DeepSeek Reasoner had positive biases (+0.242 and +0.215). A five-judge dimension-median ensemble yielded ICC(A,1)=0.454 but detected 0/5 expert-defined Safety-inadequate responses and automatically accepted 6/10 overall-inadequate responses at the fixed 4.0 cutoff. Jev matched scores yielded ICC(C,1)=0.440, ICC(A,1)=0.141, MAE=0.702 and mean signed error −0.690 against the corrected reference. Its modal Safety decisions detected 4/5 Safety-inadequate responses but falsely flagged 40–42/95 Safety-adequate responses per run. An exploratory minimal-state Jev configuration increased false acceptance from 0/10 to 7/10 in the archived comparison. CONCLUSIONS Agreement alone does not establish the validity of automated medical evaluators. Human reference reliability, signed error, false acceptance, and review burden should be considered alongside aggregate concordance. Ensemble agreement gains need not improve safety detection, and evaluator configuration changes can alter the risk profile. All screening analyses were internal and exploratory, with only five safety-inadequate cases; independent safety-enriched validation is required.