Periprosthetic joint infection (PJI) is a serious complication that can occur after joint arthroplasty, such as hip or knee replacement surgeries. It involves the invasion of the periprosthetic space by pathogens, leading to severe inflammation and often requiring complex medical intervention. PJI is associated with significant morbidity, increased healthcare costs, and a reduced quality of life for patients. This study aims to evaluate the performance of multiple supervised machine learning models in predicting PJI using clinical and demographic data collected from patients who underwent joint arthroplasty. Eight supervised machine learning models—Logistic Regression, Random Forest, XGBoost, Artificial Neural Network (ANN), k-Nearest Neighbors (KNN), AdaBoost, Gaussian Naive Bayes (GNB), and Stochastic Gradient Descent (SGD)—were trained and tested on a dataset of 27,854 patients. Models were evaluated using accuracy, precision, recall, specificity, F1 score, and area under the ROC curve (AUC). Random Forest and XGBoost showed the best overall performance, with high accuracy and balanced metrics across all evaluation criteria. KNN also performed strongly, particularly in minimizing misclassifications. GNB and SGD yielded weaker results, with higher error rates. Random Forest, XGBoost, and KNN are the most promising models for clinical implementation in PJI prediction. Their robust performance may support earlier diagnosis and improved patient outcomes in orthopedic care. This study demonstrates that machine learning models—particularly Random Forest and XGBoost—can accurately predict periprosthetic joint infection (PJI) using structured electronic health record data. By integrating these models into preoperative assessment workflows, clinicians may be able to identify high-risk patients earlier, personalize prophylactic strategies, and reduce infection-related morbidity. The implementation of these predictive tools has the potential to enhance clinical decision-making, improve surgical outcomes, and optimize the use of healthcare resources in orthopedic practice.

The evaluation of large language models (LLMs) increasingly relies on other LLMs acting as automated judges. While this approach offers scalability and efficiency, it raises serious concerns regarding evaluator reliability, positional bias, and ranking stability. This paper presents a scalable framework for diagnosing positional bias and instability in LLM-based evaluation by using controlled pairwise comparisons judged by multiple independent language models. The system supports mirrored comparisons with reversed response order, prompt injection, and surface-level perturbations (e.g., paraphrasing, lexical noise), enabling fine-grained analysis of evaluator consistency and verdict robustness. Over 3600 pairwise comparisons were conducted across five instruction-tuned open-weight models using ten open-ended prompts. The top-performing model (gemma:7b-instruct) achieved a 66.5% win rate. Evaluator agreement was uniformly high, with 100% consistency across judges, yet 48.4% of verdicts reversed under mirrored response order, indicating strong positional bias. Kendall’s Tau analysis further showed that local model rankings varied substantially across prompts, suggesting that semantic context influences evaluator judgment. All evaluation traces were stored in a graph database (Neo4j), enabling structured querying and longitudinal analysis. The proposed framework provides not only a diagnostic lens for benchmarking models but also a blueprint for fairer and more interpretable LLM-based evaluation. These findings underscore the need for structure-aware, perturbation-resilient evaluation pipelines when benchmarking LLMs. The proposed framework offers a reproducible path for diagnosing evaluator bias and ranking instability in open-ended language tasks. Future work will apply this methodology to educational assessment tasks, using rubric-based scoring and graph-based traceability to evaluate student responses in technical domains.

Background and objectives: Large language models (LLMs) show promise in automating open-ended evaluation tasks, yet their reliability in rubric-based assessment remains uncertain. Variability in scoring, feedback, and rubric adherence raises concerns about transparency and pedagogical validity in educational contexts. This study introduces CourseEvalAI, a framework designed to enhance consistency and fidelity in rubric-guided evaluation by fine-tuning a general-purpose LLM with authentic university-level instructional content. Methods: The framework employs supervised fine-tuning with Low-Rank Adaptation (LoRA) on rubric-annotated answers and explanations drawn from undergraduate computer science exams. Responses generated by both the base and fine-tuned models were independently evaluated by two human raters and two LLM judges, applying dual-layer rubrics for answers (technical or argumentative) and explanations. Inter-rater reliability was reported as intraclass correlation coefficient (ICC(2,1)), Krippendorff’s α, and quadratic-weighted Cohen’s κ (QWK), and statistical analyses included Welch’s t tests with Holm–Bonferroni correction, Hedges’ g with bootstrap confidence intervals, and Levene’s tests. All responses, scores, feedback, and metadata were stored in a Neo4j graph database for structured exploration. Results: The fine-tuned model consistently outperformed the base version across all rubric dimensions, achieving higher scores for both answers and explanations. After multiple-testing correction, only the Generative Pre-trained Transformer (GPT-4)—judged Technical Answer contrast remains statistically significant; other contrasts show positive trends without passing the adjusted threshold, and no additional significance is claimed for explanation-level results. Variance in scoring decreased, inter-model agreement increased, and evaluator feedback for fine-tuned outputs contained fewer vague or critical remarks, indicating stronger rubric alignment and greater pedagogical coherence. Inter-rater reliability analyses indicated moderate human–human agreement and weaker alignment of LLM judges to the human mean. Originality: CourseEvalAI integrates rubric-guided fine-tuning, dual-layer evaluation, and graph-based storage into a unified framework. This combination provides a replicable and interpretable methodology that enhances the consistency, transparency, and pedagogical value of LLM-based evaluators in higher education and beyond.

Background and objectives: Automated evaluation of open-ended responses remains a persistent challenge, particularly when consistency, transparency, and reproducibility are required. While large language models (LLMs) have shown promise in rubric-based evaluation, their reliability across multiple evaluators is still uncertain. Variability in scoring, feedback, and rubric adherence raises concerns about interpretability and system robustness. This study introduces GraderAssist, a graph-based, rubric-guided, multi-LLM framework designed to ensure transparent and reproducible automated evaluation. Methods: GraderAssist evaluates a dataset of 220 responses to both technical and argumentative questions, collected from undergraduate computer science courses. Six open-source LLMs and GPT-4 (as expert reference) independently scored each response using two predefined rubrics. All outputs—including scores, feedback, and metadata—were parsed, validated, and stored in a Neo4j graph database, enabling structured querying, traceability, and longitudinal analysis. Results: Cross-model analysis revealed systematic differences in scoring behavior and feedback generation. Some models produced more generous evaluations, while others aligned closely with GPT-4. Semantic analysis using Sentence-BERT embeddings highlighted distinctive feedback styles and variable rubric adherence. Inter-model agreement was stronger for technical criteria but diverged substantially for argumentative tasks. Originality: GraderAssist integrates rubric-guided evaluation, multi-model comparison, and graph-based storage into a unified pipeline. By emphasizing reproducibility, transparency, and fine-grained analysis of evaluator behavior, it advances the design of interpretable automated evaluation systems with applications in education and beyond.

Large Language Models (LLMs) are increasingly used for rubric-based assessment, but reliable automated grading requires more than a single prompt-response step. This study presents GradeAgentOps, a verification-first framework for evidence-anchored LLM exam grading that combines strict grading contracts, deterministic verification and canonicalization, bounded semantic repair, optional memory modules, and provenance-aware logging. The framework is evaluated on a university-level dataset of 1000 short open-ended exam responses from 100 students across 10 questions, with annotations from two independent expert human graders. A controlled ablation protocol compares six configurations, including a rubric-only baseline and progressively stronger variants with repair and memory augmentation. Human–human agreement provides the reference context, with overall ICC(2,1) = 0.678 and QWK = 0.678 between the two graders. Using one expert grader as the operational reference, the FULL configuration achieves the strongest model-human agreement (MAE = 1.935, RMSE = 2.500, QWK = 0.652, Within ±2 = 0.667), with the consistency memory configuration with repair (C1) emerging as the closest alternative. The gains are not uniform: improvements are more pronounced on argumentative items than on technical ones, and pairwise comparisons show that consistency memory contributes more strongly than rubric memory alone. The stronger configurations also produce cleaner accepted outputs, with lower rates of evidence-related postprocess issues, although at a moderate operational cost. Overall, within this controlled single-course evaluation setting, the results show that reliable automated grading benefits not only from model capability, but also from pipeline design choices that promote verification, evidential coherence, and stable grading behavior.

Large Language Models (LLMs) are increasingly used for rubric-based assessment, yet reliability is limited by instability, bias, and weak diagnostics. We present EvalCouncil, a committee-and-chief framework for rubric-guided grading with auditable traces and a human adjudication baseline. Our objectives are to (i) characterize domain structure in Human–LLM alignment, (ii) assess robustness to concordance tolerance and panel composition, and (iii) derive a domain-adaptive audit policy grounded in dispersion and chief–panel differences. Authentic student responses from two domains–Computer Networks (CNs) and Machine Learning (ML)–are graded by multiple heterogeneous LLM evaluators using identical rubric prompts. A designated chief arbitrator operates within a tolerance band and issues the final grade. We quantify within-panel dispersion via MPAD (mean pairwise absolute deviation), measure chief–panel concordance (e.g., absolute error and bias), and compute Human–LLM deviation. Robustness is examined by sweeping the tolerance and performing leave-one-out perturbations of panel composition. All outputs and reasoning traces are stored in a graph database for full provenance. Human–LLM alignment exhibits systematic domain dependence: ML shows tighter central tendency and shorter upper tails, whereas CN displays broader dispersion with heavier upper tails and larger extreme spreads. Disagreement increases with item difficulty as captured by MPAD, concentrating misalignment on a relatively small subset of items. These patterns are stable to tolerance variation and single-grader removals. The signals support a practical triage policy: accept low-dispersion, small-gap items; apply a brief check to borderline cases; and adjudicate high-dispersion or large-gap items with targeted rubric clarification. EvalCouncil instantiates a committee-and-chief, rubric-guided grading workflow with committee arbitration, a human adjudication baseline, and graph-based auditability in a real classroom deployment. By linking domain-aware dispersion (MPAD), a policy tolerance dial, and chief–panel discrepancy, the study shows how these elements can be combined into a replicable, auditable, and capacity-aware approach for organizing LLM-assisted grading and identifying instability and systematic misalignment, while maintaining pedagogical interpretability.