Norming of psychological tests is decisive for test score interpretation. However, conventional norming based on subgroups results either in biases or require very large samples to gather precise norms. Continuous norming methods, namely inferential, semi-parametric, and (simplified) parametric norming, propose to solve those issues. This article provides a systematic review of continuous norming. The review includes 121 publications with overall 189 studies. The main findings indicate that most studies used simplified parametric norming, not all studies considered essential distributional assumptions, and the evidence comparing different norming methods is inconclusive. In a real data example, using the standardization sample of the Need for Cognition-KIDS scale, we compared the precision of conventional, semi-parametric, and parametric norms. A hierarchy in terms of precision emerged with conventional norms being least precise, followed by semi-parametric norms, and parametric norms being most precise. We discuss these findings by comparing our findings and methods to previous studies.
13 December 2024 in Learning and Individual Differences
Psychological test manuals vary widely in their reporting of the construction and interpretation of standardized scores. Consequently, the critical evaluation of norm quality and meaning is difficult for test users and reviewers. Because a specific standard for reporting on standardized scores is lacking, we developed Guidelines for Reporting on Norm-referenced and Criterion-referenced Scores (GRoNC), following a systematic approach for creating reporting guidelines (EQUATOR). The development took place in two stages: Stage 1, developing a preliminary version of the GRoNC based on a literature review; Stage 2, a Delphi process in two rounds, involving both theoretical experts ( n = 11) and test developers ( n = 14). The GRoNC includes a series of questions and associated explanations. It supports test developers in developing and reporting upon their standardized scores, and reviewers in evaluating a psychological test on its standardized scores. We provide recommendations on using the GRoNC and conclude by describing our expectations and plans to increase the impact of the GRoNC on reporting practice.
Zusammenfassung: Normen psychometrischer Verfahren sind eine entscheidende Voraussetzung für den Vergleich und die Interpretation von Testwerten. Die konventionelle Methode der Normierung – die Bildung von Subgruppen zu kontinuierlich verteilten Merkmalen (z. B. Altersnormen) – führt jedoch zu verzerrten Normen, sofern nicht sehr große Stichproben erhoben werden. Neue kontinuierliche Normierungsmethoden wie semi-parametrisches und parametrisches Normieren scheinen eine Lösung für dieses Ökonomie-Präzisions-Dilemma darzustellen. Ziel dieses Beitrags ist es, einen systematischen Überblick über die Normierungsmethoden deutschsprachiger Verfahren mittels einer stratifizierten Stichprobe von nach 2010 veröffentlichten Verfahren aus den Bereichen Leistung, Entwicklung, Intelligenz und Persönlichkeit zu liefern. Unser systematischer Überblick zeigt, dass die angewandten Normierungspraktiken weitestgehend den Teststandards entsprechen. Mehrheitlich werden konventionelle Normen berichtet, wobei sich ebenfalls ein Trend hin zur Nutzung kontinuierlicher Normierungsmethoden zeigt. Optimierbar erscheinen die Beschreibung des Normierungsprozesses und die Formulierung von Anwendungsempfehlungen. Beides wird jedoch gebraucht, um die Nutzung kontinuierlicher Normierungsmethoden bei der Testentwicklung zu erleichtern.
Remote intelligence testing has multiple advantages, but cheating is possible without proper supervision. Proctoring aims to address this shortcoming, yet prior research on its effects has primarily investigated reasoning tasks, in which cheating is generally difficult. This study provides an overview of recent research on the effects of proctoring and on studies in intelligence test settings. Moreover, we conducted an empirical study testing the effects of webcam-based proctoring with a multidimensional intelligence test measuring reasoning, short-term memory, processing speed, and divergent thinking. The study was conducted in a low-stakes context, with participants receiving a fixed payment regardless of performance. Participants completed the test under proctored (n = 74, webcam consent), unproctored random (n = 75, webcam consent), or unproctored chosen (n = 77, no webcam consent) conditions. Scalar measurement invariance was observed for reasoning, processing speed, and divergent thinking, but not for memory. Proctoring had no significant main effect on test performance but showed a significant interaction with test type. Proctored participants outperformed the unproctored chosen group significantly in divergent thinking and scored descriptively higher in reasoning and processing speed, but slightly lower in memory. Observable cheating under proctored conditions was rare (4%), mostly involving note-taking or photographing the screen. We conclude that proctoring is crucial for easily cheatable tasks, such as memory tasks, but currently less critical for complex cognitive tasks.
The MAGMA-package offers nearest neighbor matching for two to four groups. It also includes the option to match data of a 2x2 Design. In addition, MAGMA includes a framework for evaluating the post-matching balance. This vignette is a tutorial on MAGMA. It demonstrates the main MAGMA function on an included simulated dataset. In total, this vignette includes a two-group, a three-group, and a 2x2-design which is approximately equivalent to a four-group example. This three examples are conducted in a standard matching process, as well as with exact matching. Parts of this tutorial are also included in the respective help pages of the functions. For details of the algorithm and a more extensive introduction of MAGMA, see Feuchter et al. (2022).