This study is a methodological-substantive synergy, demonstrating the power and flexibility of exploratory structural equation modeling (ESEM) methods that integrate confirmatory and exploratory factor analyses (CFA and EFA), as applied to substantively important questions based on multidimentional students' evaluations of university teaching (SETs). For these data, there is a well established ESEM structure but typical CFA models do not fit the data and substantially inflate correlations among the nine SET factors (median rs = .34 for ESEM, .72 for CFA) in a way that undermines discriminant validity and usefulness as diagnostic feedback. A 13-model taxonomy of ESEM measurement invariance is proposed, showing complete invariance (factor loadings, factor correlations, item uniquenesses, item intercepts, latent means) over multiple groups based on the SETs collected in the first and second halves of a 13-year period. Fully latent ESEM growth models that unconfounded measurement error from communality showed almost no linear or quadratic effects over this 13-year period. Latent multiple indicators multiple causes models showed that relations with background variables (workload/difficulty, class size, prior subject interest, expected grades) were small in size and varied systematically for different ESEM SET factors, supporting their discriminant validity and a construct validity interpretation of the relations. A new approach to higher order ESEM was demonstrated, but was not fully appropriate for these data. Based on ESEM methodology, substantively important questions were addressed that could not be appropriately addressed with a traditional CFA approach.

In multilevel modeling (MLM), group-level (L2) characteristics are often measured by aggregating individual-level (L1) characteristics within each group so as to assess contextual effects (e.g., group-average effects of socioeconomic status, achievement, climate). Most previous applications have used a multilevel manifest covariate (MMC) approach, in which the observed (manifest) group mean is assumed to be perfectly reliable. This article demonstrates mathematically and with simulation results that this MMC approach can result in substantially biased estimates of contextual effects and can substantially underestimate the associated standard errors, depending on the number of L1 individuals per group, the number of groups, the intraclass correlation, the sampling ratio (the percentage of cases within each group sampled), and the nature of the data. To address this pervasive problem, the authors introduce a new multilevel latent covariate (MLC) approach that corrects for unreliability at L2 and results in unbiased estimates of L2 constructs under appropriate conditions. However, under some circumstances when the sampling ratio approaches 100%, the MMC approach provides more accurate estimates. Based on 3 simulations and 2 real-data applications, the authors evaluate the MMC and MLC approaches and suggest when researchers should most appropriately use one, the other, or a combination of both approaches.

This article is a methodological-substantive synergy. Methodologically, we demonstrate latent-variable contextual models that integrate structural equation models (with multiple indicators) and multilevel models. These models simultaneously control for and unconfound measurement error due to sampling of items at the individual (L1) and group (L2) levels and sampling error due the sampling of persons in the aggregation of L1 characteristics to form L2 constructs. We consider a set of models that are latent or manifest in relation to sampling items (measurement error) and sampling of persons (sampling error) and discuss when different models might be most useful. We demonstrate the flexibility of these 4 core models by extending them to include random slopes, latent (single-level or cross-level) interactions, and latent quadratic effects. Substantively we use these models to test the big-fish-little-pond effect (BFLPE), showing that individual student levels of academic self-concept (L1-ASC) are positively associated with individual level achievement (L1-ACH) and negatively associated with school-average achievement (L2-ACH)—a finding with important policy implications for the way schools are structured. Extending tests of the BFLPE in new directions, we show that the nonlinear effects of the L1-ACH (a latent quadratic effect) and the interaction between gender and L1-ACH (an L1 × L1 latent interaction) are not significant. Although random-slope models show no significant school-to-school variation in relations between L1-ACH and L1-ASC, the negative effects of L2-ACH (the BFLPE) do vary somewhat with individual L1-ACH. We conclude with implications for diverse applications of the set of latent contextual models, including recommendations about their implementation, effect size estimates (and confidence intervals) appropriate to multilevel models, and directions for further research in contextual effect analysis.

The analysis of factor structures is one of the most critical psychometric applications. Frequently, variables (i.e., items or indicators) resulting from questionnaires using ordinal items with 2–7 categories are used. There are plenty of articles that recommend treating ordinal variables in a factor analysis by default as ordinal and not as continuous imposing a multivariate normal distribution assumption. In this article, we exhibit that the reasoning behind such suggestions is flawed. In our view, findings from simulation studies cannot tell about the right modeling strategy of ordinal variables in factor analysis. Moreover, it is argued that ordinal factor models impose a normality assumption for underlying continuous variables, which might also often be incorrect in empirical applications. However, researchers seldom opt for more flexible modeling strategies that involve correctly specified distributions. Finally, the consequences of modeling choices for validity, reliability, measurement invariance, handling of missing data, and the assessment of global model fit are discussed.

Zusammenfassung. Fehlende Werte stellen in der empirisch-psychologischen Forschung oftmals ein Problem dar. Häufig verwendete Verfahren wie fallweiser und paarweiser Ausschluss, Regression- und Mean-Imputation sind aus methodischer Sicht defizitär. Alternative Verfahren für die Analyse von Datensätzen mit fehlenden Werten, die in den letzten drei Jahrzehnten entwickelt wurden, werden in der Forschungspraxis noch selten angewendet. Der vorliegende Beitrag führt zunächst in die grundlegende Terminologie fehlender Werte nach Rubin (1976) ein. Im Anschluss daran wird eine Übersicht der in der Literatur diskutierten Ansätze zum Umgang mit fehlenden Werten vorgenommen, wobei drei Typen von Verfahren unterschieden werden: klassische Verfahren (z.B. fallweiser Ausschluss), imputationsbasierte Verfahren, in denen fehlende Werte ersetzt (imputiert) werden, und modellbasierte Verfahren, in denen die Schätzung des Modells und die Behandlung der fehlenden Werte in einem Schritt vorgenommen werden. Anhand eines Datenbeispiels wird dann der Einsatz der Multiplen Imputation veranschaulicht. Abschließend werden Implikationen für die Forschungspraxis diskutiert.