15 December 2021 in Behavior Research Methods
In social sciences, the study of group differences concerning latent constructs is ubiquitous. These constructs are generally measured by means of scales composed of ordinal items. In order to compare these constructs across groups, one crucial requirement is that they are measured equivalently or, in technical jargon, that measurement invariance (MI) holds across the groups. This study compared the performance of scale- and item-level approaches based on multiple group categorical confirmatory factor analysis (MG-CCFA) and multiple group item response theory (MG-IRT) in testing MI with ordinal data. In general, the results of the simulation studies showed that MG-CCFA-based approaches outperformed MG-IRT-based approaches when testing MI at the scale level, whereas, at the item level, the best performing approach depends on the tested parameter (i.e., loadings or thresholds). That is, when testing loadings equivalence, the likelihood ratio test provided the best trade-off between true-positive rate and false-positive rate, whereas, when testing thresholds equivalence, theχ2test outperformed the other testing strategies. In addition, the performance of MG-CCFA’s fit measures, such as RMSEA and CFI, seemed to depend largely on the length of the scale, especially when MI was tested at the item level. General caution is recommended when using these measures, especially when MI is tested for each item individually.
16 May 2022 in Educational and Psychological Measurement
Assessing the measurement model (MM) of self-report scales is crucial to obtain valid measurements of individuals’ latent psychological constructs. This entails evaluating the number of measured constructs and determining which construct is measured by which item. Exploratory factor analysis (EFA) is the most-used method to evaluate these psychometric properties, where the number of measured constructs (i.e., factors) is assessed, and, afterward, rotational freedom is resolved to interpret these factors. This study assessed the effects of an acquiescence response style (ARS) on EFA for unidimensional and multidimensional (un)balanced scales. Specifically, we evaluated (a) whether ARS is captured as an additional factor, (b) the effect of different rotation approaches on the content and ARS factors recovery, and (c) the effect of extracting the additional ARS factor on the recovery of factor loadings. ARS was often captured as an additional factor in balanced scales when it was strong. For these scales, ignoring extracting this additional ARS factor, or rotating to simple structure when extracting it, harmed the recovery of the original MM by introducing bias in loadings and cross-loadings. These issues were avoided by using informed rotation approaches (i.e., target rotation), where (part of) the rotation target is specified according to a priori expectations on the MM. Not extracting the additional ARS factor did not affect the loading recovery in unbalanced scales. Researchers should consider the potential presence of ARS when assessing the psychometric properties of balanced scales and use informed rotation approaches when suspecting that an additional factor is an ARS factor.
25 January 2023 in PsyArXiv
LBLucca Michelle Brandner
We studied the Amsterdam Sexual Pleasure Inventory’s (ASPI, Vol. 1.0) psychometric properties to present evidence regarding its intended interpretation and use. The ASPI is a theory-based and revised self-report battery which aims to assess different domains of state and trait sexual pleasure in survey-research in gender-diverse, sex-diverse, and relationship-diverse populations. We collected quantitative (n = 1371) and qualitative data (n = 637) using a cross-sectional multi-method design targeting the general (German-speaking) population. The theory-based 5-factor ESEM showed good and the PCA of the two general exploratory index-scales showed acceptable structural validity evidence. Measurement invariance of the 5-factor models was given for male and female (assigned-at-birth) participants and for sexually functional-scoring and dysfunctional-scoring people. Coefficient omega indicated that all scales, except those of one facet, showed acceptable to very good internal consistency reliability. The ASPI’s convergent and discriminant associations with sexological and psychological constructs demonstrated good overall construct validity evidence and the scales showed differential utility in differentiating known-groups. Participants understood the items as intended and felt that the ASPI covers relevant facets of sexual pleasure. The ASPI might help understand how individuals differ in experiencing sexual pleasure and how different contexts enable some people to experience pleasure while disadvantaging others.
12 March 2025 in PsyArXiv
This article builds and extends on recent work using Large Language Models (LLMs) in psychometrics to generate pseudo-discrimination parameters. While earlier work looked at pseudo-discrimination on an item-by-construct basis, we introduce pseudo-factor analysis to enhance scale design. Pseudo-factor analysis is a data-less, model-based approach to evaluating aspects of a latent construct’s measurement model, such as dimensionality and the relations between factors and their indicators. Across two studies using Five and Six-factor personality frameworks, a variety of sentence transformer models, and three encoding approaches (i.e., atomic, atomic reversed, and macro), pseudo-factor analyses recovered theoretically expected structures. These pseudo-factor structures were strongly related to their established empirical factor structures based on factor analyses of human ratings in prior published research. We suggest that Pseudo-Factor Analysis is a viable method for checking and potentially modifying scale items after item generation and before item trialing. We provide a Shiny application for calculating pseudo-factor analysis parameters and related psychometric estimates.
2 November 2023 in Structural Equation Modeling: A Multidisciplinary Journal