Holding model misspecification constant, the behavior of fit indices depends on factors such as the number of variables being modeled (model size), and the average observed correlation (magnitude of factor loadings or measurement quality). We examine by simulation the interplay of these factors with sample size in CFA models. When a biased estimator of the fit index is used (CFI, TLI, or GFI), the behavior of the sample indices depends on sample size, rendering establishing cutoff values impossible. When an unbiased estimator is used (SRMR, or RMSEA) the behavior of the indices matches that of the population parameter and depends on the average R2 of the observed variables (communality); and for the RMSEA, also on model size. The use of the unbiased SRMR with a cutoff value adjusted by R2 is recommended as it enables assessing the degree of a model misspecification across model size, sample size, and measurement quality.
1 December 1998 in Journal of Educational Measurement
The goal of this study was to assess the feasibility of an approach to adaptive testing based on item models. A simulation study was designed to explore the affects of item modeling on score precision and bias, and two experimental tests were administered — an experimental, on‐the‐fly, adaptive quantitative‐reasoning test as well as a linear test. Results of the simulation study showed that under different levels of isomorphicity, there was no bias, but precision of measurement was eroded, especially in the middle range of the true‐score scale. However, the comparison of adaptive test scores with operational Graduate Record Examinations (GRE) test scores matched the test‐retest correlation observed under operational conditions. Analyses of item functioning on linear forms suggested a high level of isomorphicity across items within models. The current study provides a promising first step toward significant cost and theoretical improvement in test creation methodology for educational assessment.
Psychometric properties of computerized testing, together with anxiety and comfort of examinees, are investigated in relation to item selection routine and the opportunity for response review. Two different hypotheses involving examinee anxiety were used to design test properties: perceived control and perceived performance. The study involved three types of administration of a computerized English test for Spanish speakers (adaptive, easy adaptive, and fixed) and four review conditions (no review, review at end, review by blocks of 5 items, and review item-by-item). These were applied to a sample of 557 first-year psychology undergraduate students to examine main and interaction effects of test type and review on psychometric and psychological variables. Statistically significant effects were found in test precision among the different types of test. Response review improved ability estimates and increased testing time. No psychological effects on anxiety were found. Examinees in all review conditions considered more important the possibility of review than those who were not allowed to review. These results concur with previous findings on examinees' preference for item review and raise some issues that should be addressed in the field of tests with item review.
This article introduces Bayesian estimation and evaluation procedures for the multidimensional nominal response model. The utility of this model is to perform a nominal factor analysis of items that consist of a finite number of unordered response categories. The key aspect of the model, in comparison with traditional factorial model, is that there is a factor loading for each response category on the latent traits, instead of having one factor loadings associated to the items. The extended parameterization of the multidimensional nominal response model requires large samples for estimation. When sample size is of a moderate or small size, some of these parameters may be weakly empirically identifiable and the estimation algorithm may run into difficulties. We propose a Bayesian MCMC inferential algorithm to estimate the parameters and the number of latent traits underlying the multidimensional nominal response model. Two Bayesian approaches to model evaluation were compared: discrepancy statistics (DIC, WAICC and LOO) that provide an indication of the relative merit of different models, and the standardized generalized discrepancy measure that requires resampling data and is computationally more involved. A simulation study was conducted to compare these two approaches, and the results show that the standardized generalized discrepancy measure can be used to reliably estimate the dimensionality of the model whereas the discrepancy statistics are questionable. The paper also includes an example with real data in the context of learning styles, in which the model is used to conduct an exploratory factor analysis of nominal data.