Psychological science requires reliable measures. Within systematic literature reviews, reliability hinges on high interrater agreement during data extraction. Yet, the extraction process has been time-consuming. Efforts to accelerate the process using technology have shown limited success until generative artificial intelligence (genAI), particularly large language models (LLMs), accurately extracted variables from medical studies. Nonetheless, for psychological researchers, it remains unclear how to utilize genAI for data extraction, given the range of tested variables, the medical context, and the variability in accuracy. We systematically assessed extraction accuracy and error patterns across domains in psychology by comparing genAI-extracted and human-extracted data from 22 systematic review databases published in the Psychological Bulletin. Eight LLMs extracted 312,329 data points from 2,179 studies on 186 variables. LLM extractions achieved unacceptable accuracy on all metrics for 20% of variables. For 46% of variables, accuracy was acceptable for some metrics and unacceptable for others. LLMs reached acceptable but not high accuracy on all metrics in 15%, high but not excellent in 8%, and excellent accuracy in 12% of variables. Accuracy varied most between variables, less between systematic reviews, and least between LLMs. Moderator analyses using a hierarchical logistic regression, hierarchical linear model, and meta-analysis revealed that accuracy was higher for variables describing studies' context and moderator variables compared to variables for effect size calculation. Also, accuracy was higher in systematic reviews with more detailed variable descriptions and positively correlated with model sizes. We discuss directions for investigating ways to use genAI to accelerate data extractions while ensuring meaningful human control. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
Self-assessment accuracy is crucial for effective learning and performance. However, learners tend to be inaccurate in their self-assessment, calling for effective interventions to increase their accuracy. Over the past three decades, research has shown that providing feedback can guide and improve self-assessment accuracy. However, the magnitude and heterogeneity of feedback effects on self-assessment accuracy have not been meta-analytically investigated. To address this research desiderate, we conducted a systematic review and a three-level meta-analysis with random effects to quantify the overall feedback effect and to examine potential moderator effects (characteristics of feedback, self-assessment, sample, and task). A total of 201 effect sizes involving 5,487 participants from 47 studies were included in the analysis. We found a significant overall effect (g = 0.34), indicating a positive impact of feedback on self-assessment accuracy. This effect was significantly moderated by the type of feedback information, where we found the largest effect for Knowledge of Result feedback (KR), followed by elaborated feedback (EF), Knowledge of Correct Response feedback (KCR), and feedback standards. Additionally, significant moderation effects were identified relating to self-assessment and sample characteristics. As a practical implication, we conclude that providing feedback can be an effective intervention to support self-assessment accuracy.
By quantitatively integrating results of meta-analyses, second-order meta-analyses (SOMAs) from educational psychology address far-reaching questions of interest to adjacent scientific fields, policy, and educational practice. Conducting a SOMA, however, is time-consuming, particularly the extraction of statistical results from meta-analyses. Large Language Models (LLMs) have the potential to accelerate data extraction, as evidenced by evaluations of data extraction from primary studies. Evaluations of their accuracy in extracting results from meta-analyses to conduct SOMA are needed. We compare the accuracy of three LLMs with that of human experts extracting data from 156 educational meta-analyses investigating students’ achievement, randomly chosen from the Visible Learning database. LLMs and humans extracted effect sizes, the number of included studies, and the number of students. To assess accuracy, we established a gold standard benchmark by resolving discrepancies between LLMs and human experts through expert adjudication. Results show that the distributions of data from all coders and the standard were similar and showed no systematic bias. Accuracy reached ICCs of 0.95/0.81 for the two human experts, and 0.96/0.97/0.96 for LLMs (Gemini2.5Pro/GPT4.1/GPTo3), with percentage agreement of 86%/80% (humans) and 81%/78%/77% (LLMs). The results demonstrate that LLMs achieve data extraction accuracy comparable to that of human experts. We discuss the conditions under which data extracted by LLMs, humans, or a hybrid of both can be considered validated for use in educational SOMAs.