One major challenge of longitudinal data analysis is to find an appropriate statistical model that corresponds to the theory of change and the research questions at hand. In the present article, we argue that continuous-time models are well suited to study the continuously developing constructs of primary interest in the education sciences and outline key advantages of using this type of model. Furthermore, we propose the continuous-time latent curve model with structured residuals (CT-LCM-SR) as a suitable model for many research questions in the education sciences. The CT-LCM-SR combines growth and dynamic modeling and thus provides descriptions of both trends and process dynamics. We illustrate the application of the CT-LCM-SR with data from PISA reading literacy assessment of 2000 to 2018 and provide a tutorial and annotated code for setting up the CT-LCM-SR model.
13 September 2024 in International Journal of Artificial Intelligence in Education
Recent investigations in automated essay scoring research imply that hybrid models, which combine feature engineering and the powerful tools of deep neural networks (DNNs), reach state-of-the-art performance. However, most of these findings are from holistic scoring tasks. In the present study, we use a total of four prompts from two different corpora consisting of both L1 and L2 learner essays annotated with trait scores (e.g., content, organization, and language quality). In our main experiments, we compare three variants of trait-specific models using different inputs: (1) models based on 220 linguistic features, (2) models using essay-level contextual embeddings from the distilled version of the pre-trained transformer BERT (DistilBERT), and (3) a hybrid model using both types of features. Results imply that when trait-specific models are trained based on a single resource, the feature-based models slightly outperform the embedding-based models. These differences are most prominent for the organization traits. The hybrid models outperform the single-resource models, indicating that linguistic features and embeddings indeed capture partially different aspects relevant for the assessment of essay traits. To gain more insights into the interplay between both feature types, we run addition and ablation tests for individual feature groups. Trait-specific addition tests across prompts indicate that the embedding-based models can most consistently be enhanced in content assessment when combined with morphological complexity features. Most consistent performance gains in the organization traits are achieved when embeddings are combined with length features, and most consistent performance gains in the assessment of the language traits when combined with lexical complexity, error, and occurrence features. Cross-prompt scoring again reveals slight advantages for the feature-based models.
Psychological science requires reliable measures. Within systematic literature reviews, reliability hinges on high interrater agreement during data extraction. Yet, the extraction process has been time-consuming. Efforts to accelerate the process using technology have shown limited success until generative artificial intelligence (genAI), particularly large language models (LLMs), accurately extracted variables from medical studies. Nonetheless, for psychological researchers, it remains unclear how to utilize genAI for data extraction, given the range of tested variables, the medical context, and the variability in accuracy. We systematically assessed extraction accuracy and error patterns across domains in psychology by comparing genAI-extracted and human-extracted data from 22 systematic review databases published in the Psychological Bulletin. Eight LLMs extracted 312,329 data points from 2,179 studies on 186 variables. LLM extractions achieved unacceptable accuracy on all metrics for 20% of variables. For 46% of variables, accuracy was acceptable for some metrics and unacceptable for others. LLMs reached acceptable but not high accuracy on all metrics in 15%, high but not excellent in 8%, and excellent accuracy in 12% of variables. Accuracy varied most between variables, less between systematic reviews, and least between LLMs. Moderator analyses using a hierarchical logistic regression, hierarchical linear model, and meta-analysis revealed that accuracy was higher for variables describing studies' context and moderator variables compared to variables for effect size calculation. Also, accuracy was higher in systematic reviews with more detailed variable descriptions and positively correlated with model sizes. We discuss directions for investigating ways to use genAI to accelerate data extractions while ensuring meaningful human control. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
Croon and van Veldhoven discussed a model for analyzing micro–macro multilevel designs in which a variable measured at the upper level is predicted by an explanatory variable that is measured at the lower level. Additionally, the authors proposed an approach for estimating this model. In their approach, estimation is carried out by running a regression analysis on Bayesian Expected a Posterior (EAP) estimates. In this article, we present an extension of this approach to interaction and quadratic effects of explanatory variables. Specifically, we define the Bayesian EAPs, discuss a way for estimating them, and we show how their estimates can be used to obtain the interaction and the quadratic effects. We present the results of a “proof of concept” via Monte Carlo simulation, which we conducted to validate our approach and to compare two resampling procedures for obtaining standard errors. Finally, we discuss limitations of our proposed extended Bayesian EAP-based approach.
3 May 2023 in Structural Equation Modeling: A Multidisciplinary Journal
The recently proposed continuous-time latent curve model with structured residuals (CT-LCM-SR) addresses several challenges associated with longitudinal data analysis in the behavioral sciences. First, it provides information about process trends and dynamics. Second, using the continuous-time framework, the CT-LCM-SR can handle unequally spaced measurement occasions and describes processes independently of the length of the time intervals used in a given study. Third, it is a hierarchical model. Thus, multiple subjects can be analyzed simultaneously. However, subjects might also differ in dynamics and trends. Therefore, in the present paper, we extend the CT-LCM-SR to capture these differences as well. This newly proposed random coefficients continuous-time latent curve model with structured residuals (RC-CT-LCM-SR) is introduced theoretically and technically. Additionally, we provide an illustrative example with data from the Health and Retirement Study (HRS), and we show how the RC-CT-LCM-SR can be used to study multiple sources of between-subject differences over time.
1 August 2026 in Journal of Educational Psychology