6 June 2024 in Journal of Educational Measurement
23 January 2026 in Psychometrika
Various item selection algorithms have been proposed for cognitive diagnostic computerized adaptive testing (CD-CAT), with the goal of efficiently diagnosing examinees’ strengths and weaknesses. However, these algorithms often come with significant computational costs, which can hinder their practical implementation. A likelihood-based profile shrinkage (LBPS) algorithm is proposed to simplify the item selection process and reduce the computational cost in CD-CAT. Our simulation results indicate that incorporating LBPS into existing item selection methods yields substantial computational efficiency gains, with greater reductions in computation time as the number of attributes and test length increase. Additionally, LBPS maintains estimation accuracy at both the attribute and pattern levels. These findings suggest that LBPS is a scalable and effective solution for the item selection of CD-CAT in complex scenarios.
23 April 2026 in PsyArXiv
Large language models (LLMs) are increasingly used to generate synthetic data for research and evaluation, including simulating student responses to educational assessments. However, their validity for emulating authentic human performance remains unclear. This study evaluates whether LLMs can-generate assessment data that preserve the score distribution, subgroup relationships, item functioning, latent-variable model properties, and overall distributional similarity required for valid measurement. Using fourth-grade mathematics data from TIMSS 2011, we compare synthetic responses generated by GPT-4o and GPT-5 under multiple prompting conditions against empirical student responses and theory-driven simulated data from a cognitive diagnostic model (CDM). Across both LLM architectures, LLM-generated data exhibit systematic score inflation and variance compression, with distortions more pronounced for GPT-5. Item easiness is overestimated, item discriminations are attenuated, and alignment with empirical item properties is weak. Adding information on cognitive mastery profiles and personal background modestly increases variability but does not recover human-like psychometric structure. Although the CDM shows adequate fit to some LLM-generated datasets, item parameter recovery remains poor. In contrast, CDM-based simulations closely reproduce human score distributions, item properties, and latent structure. These results suggest that LLM-generated synthetic assessment data deviate systematically from human data, which limits their usefulness for inferences requiring structural fidelity. Caution is therefore warranted when using LLM-generated synthetic data in measurement contexts where variance, item functioning, and latent structure are central.