Background and objectives: Automated evaluation of open-ended responses remains a persistent challenge, particularly when consistency, transparency, and reproducibility are required. While large language models (LLMs) have shown promise in rubric-based evaluation, their reliability across multiple evaluators is still uncertain. Variability in scoring, feedback, and rubric adherence raises concerns about interpretability and system robustness. This study introduces GraderAssist, a graph-based, rubric-guided, multi-LLM framework designed to ensure transparent and reproducible automated evaluation. Methods: GraderAssist evaluates a dataset of 220 responses to both technical and argumentative questions, collected from undergraduate computer science courses. Six open-source LLMs and GPT-4 (as expert reference) independently scored each response using two predefined rubrics. All outputs—including scores, feedback, and metadata—were parsed, validated, and stored in a Neo4j graph database, enabling structured querying, traceability, and longitudinal analysis. Results: Cross-model analysis revealed systematic differences in scoring behavior and feedback generation. Some models produced more generous evaluations, while others aligned closely with GPT-4. Semantic analysis using Sentence-BERT embeddings highlighted distinctive feedback styles and variable rubric adherence. Inter-model agreement was stronger for technical criteria but diverged substantially for argumentative tasks. Originality: GraderAssist integrates rubric-guided evaluation, multi-model comparison, and graph-based storage into a unified pipeline. By emphasizing reproducibility, transparency, and fine-grained analysis of evaluator behavior, it advances the design of interpretable automated evaluation systems with applications in education and beyond.

Large Language Models (LLMs) are increasingly used for rubric-based assessment, yet reliability is limited by instability, bias, and weak diagnostics. We present EvalCouncil, a committee-and-chief framework for rubric-guided grading with auditable traces and a human adjudication baseline. Our objectives are to (i) characterize domain structure in Human–LLM alignment, (ii) assess robustness to concordance tolerance and panel composition, and (iii) derive a domain-adaptive audit policy grounded in dispersion and chief–panel differences. Authentic student responses from two domains–Computer Networks (CNs) and Machine Learning (ML)–are graded by multiple heterogeneous LLM evaluators using identical rubric prompts. A designated chief arbitrator operates within a tolerance band and issues the final grade. We quantify within-panel dispersion via MPAD (mean pairwise absolute deviation), measure chief–panel concordance (e.g., absolute error and bias), and compute Human–LLM deviation. Robustness is examined by sweeping the tolerance and performing leave-one-out perturbations of panel composition. All outputs and reasoning traces are stored in a graph database for full provenance. Human–LLM alignment exhibits systematic domain dependence: ML shows tighter central tendency and shorter upper tails, whereas CN displays broader dispersion with heavier upper tails and larger extreme spreads. Disagreement increases with item difficulty as captured by MPAD, concentrating misalignment on a relatively small subset of items. These patterns are stable to tolerance variation and single-grader removals. The signals support a practical triage policy: accept low-dispersion, small-gap items; apply a brief check to borderline cases; and adjudicate high-dispersion or large-gap items with targeted rubric clarification. EvalCouncil instantiates a committee-and-chief, rubric-guided grading workflow with committee arbitration, a human adjudication baseline, and graph-based auditability in a real classroom deployment. By linking domain-aware dispersion (MPAD), a policy tolerance dial, and chief–panel discrepancy, the study shows how these elements can be combined into a replicable, auditable, and capacity-aware approach for organizing LLM-assisted grading and identifying instability and systematic misalignment, while maintaining pedagogical interpretability.

Large language models are increasingly used as automated graders, yet their reliability under answer-side manipulation and their behavior in multi-model panels remain insufficiently understood. This paper introduces EvalHack, a matrix benchmark in which a fixed committee of four LLMs grades university-level machine learning exam answers under a strict integer-only contract (0–10) grounded in instructor-authored rubric artifacts. The dataset comprises 100 students answering 10 short, open-ended items (1000 answers). For each answer, the evaluation includes a clean version and two content-preserving adversarial variants that operate only on the student text: A₁, a visible coercive suffix appended to the answer, and A₂, a stealth variant that uses Unicode control characters (e.g., zero-width and bidirectional marks) to embed an instruction. EvalHack instruments the full grading pipeline, recording item-level member scores, the committee aggregate, within-panel disagreement, and discrepancies to human grades. Empirically, answer-side edits induce systematic score inflation and stronger top-end concentration, with edited answers clustering near the upper end of the scale. Within-panel disagreement, measured as the range between the highest and lowest member score, varies across conditions, with median Consistency Spread values of 3.0 (clean), 2.0 (A₁), and 6.0 (A₂). Compared to human graders, the panel is more lenient on average (MAE = 1.897; bias human − panel = −1.345). Finally, grouping items by disagreement shows that low-disagreement items exhibit smaller human-panel errors, indicating that within-panel spread can serve as a practical uncertainty signal for routing difficult answers to human review or to larger/more specialized panels.

Large language models are increasingly used for automated grading, but final-score agreement does not reveal whether credit is assigned to the rubric element affected by a response change. This study introduces CreditTrace-LLM, a controlled framework for auditing directional responsiveness, credit locality, paraphrase stability, and correspondence with expert evaluations in rubric-guided grading. The evaluation used 500 response families across ten questions, equally divided between technical and argumentative tasks. Each family included a baseline response (C0), a positive intervention adding rubric-relevant evidence (C+), a negative intervention removing or weakening such evidence (C−), and a placebo paraphrase (CP) constructed through lexical or syntactic reformulation with the intention of preserving rubric-relevant meaning. Eight local open-weight LLM graders produced 15,999 structurally valid Gold-point-level outputs from 16,000 expected evaluations. Targeted Gold-point scores changed in the expected direction in 75.7% of C+ comparisons and 50.2% of C− comparisons. Localized directional success was lower under C+ and C−, at 31.5% and 21.3%, respectively, indicating frequent non-target score changes. Under CP, total-score stability was 75.4%, while complete-profile stability was 73.6%. Direction concordance with the mean expert score change was 69.8% for C+, 47.4% for C−, and 18.2% for CP. The CP condition also showed substantial variability in expert scoring, particularly for argumentative responses, indicating that intended rubric-relevant meaning preservation did not guarantee score invariance. Final-score behavior alone is insufficient for validating rubric-guided LLM grading. CreditTrace-LLM therefore evaluates target responsiveness, credit locality, paraphrase stability, and traceable Gold-point-level outputs as complementary diagnostic dimensions rather than as predefined criteria for acceptable grading performance.

Local open-weight large language models (LLMs) are increasingly used in privacy-sensitive settings, yet isolated prompts may not reveal whether safety boundaries remain stable during conversation. SafeBoundary-LLM evaluated seven local models across 14 sensitive domains, 84 boundary sets, 672 single-turn prompts, and 84 five-turn escalation conversations; the same models were evaluated separately on XSTest and JBB-Behaviors. Evaluators R1 and R2 independently classified all 12,194 responses, with R2 labels used for primary outcomes and unreconciled labels used for reliability analysis. Exact agreement exceeded 93% in each dataset. In SafeBoundary-LLM, 456 out of 7644 responses (5.97%) were confirmed-or-mixed failures. The multi-turn failure rate was 14.69% versus 0.51% for single-turn prompts, yielding a rate ratio of 28.80 (95% CI [21.20, 43.80]; Holm-adjusted p = 0.0006); boundary collapse occurred only at Turns 4–5, and role-play bypass accounted for 299 out of 456 failures. On answer-expected items, over-refusal was 4.34% in XSTest and 17.29% in JBB-Behaviors, whereas unsafe compliance on refusal-expected items was 0.36% and 0.86%, respectively. These findings support an evaluation strategy that includes public single-turn benchmarks, controlled multi-turn escalation, independent human review, and traceable audit records for locally deployed LLMs.

Background: Multi-agent large language model (LLM) systems increasingly rely on structured interaction, debate, and final model-based evaluation, yet an unsupported causal attribution introduced by one participant may propagate through the interaction and influence subsequent decisions. This study introduces ClaimArena-LLM, a controlled framework for auditing adversarial propagation, evidentiary provenance, and final-stage evaluator vulnerability in multi-agent causal reasoning. Methods: ClaimArena-LLM uses 100 synthetic causal incident cases with predefined evidence-supported and false causal propositions. Each case was evaluated under matched adversarial and all-honest debate conditions using four functional roles: investigator, honest analyst, adversarial analyst, and auditor. The experimental program comprised 1460 validated runs, including the 200-run main experiment, claim-source lineage visibility and debate-round ablations, an 800-run balanced model–role rotation, and a 60-run proposition-identifier and presentation-order sensitivity control. Outcomes were scored deterministically from structured causal propositions, evidence references, claim lineage, reports, and auditor verdicts. Results: The adversarial analyst maintained the predefined false causal target with 97.6% mean fidelity. Adversarial exposure increased investigator shifts toward the false target by 16.0 percentage points (Holm-adjusted p = 0.035). At the auditor stage, false-target attribution increased by 11.0 percentage points (adjusted p = 0.043), verdict contamination by 20.0 percentage points (adjusted p = 0.002), and semantic provenance validity decreased by 20.0 percentage points (adjusted p = 0.002). Neither the manipulation nor the false target was explicitly detected. The ablations showed no robust adjusted differences, model–role rotation revealed substantial configuration dependence, and no benchmark-characteristic interaction remained significant after Holm correction. The proposition-identifier and presentation-order sensitivity control showed no statistically robust evidence that fixed proposition encoding explained the observed condition contrasts. Conclusions: Adversarial causal influence in multi-agent LLM systems is selective rather than uniformly propagative and depends strongly on functional role and model–role configuration. Within the tested architecture, a final LLM auditor cannot be assumed to provide an independent safeguard against upstream contamination. Reliable evaluation therefore requires separate assessment of causal correctness, propagation, semantic provenance, and final-stage adversarial awareness rather than reliance on final answer accuracy alone.