1 February 2014 in Psicothema
Validity evidence based on the internal structure of an assessment is one of the five forms of validity evidence stipulated in the Standards for Educational and Psychological Testing of the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. In this paper, we describe the concepts underlying internal structure and the statistical methods for gathering and analyzing internal structure. An in-depth description of the traditional and modern techniques for evaluating the internal structure of an assessment. Validity evidence based on the internal structure of an assessment is necessary for building a validity argument to support the use of a test for a particular purpose. The methods described in this paper provide practitioners with a variety of tools for assessing dimensionality, measurement invariance and reliability for an educational test or other types of assessment.
8 January 2016 in Journal of Research in Science Teaching
3 October 2016 in International Journal of Testing
When examinees' test-taking motivation is questionable, practitioners must determine whether careless responding is of practical concern and if so, decide on the best approach to filter such responses. As there has been insufficient research on these topics, the objectives of this study were to: a) evaluate the degree of underestimation in the true mean when careless responses are present, and b) compare the effectiveness of two filtering procedures in purifying biased aggregated-scores. Results demonstrated that: a) the true mean was underestimated by around 0.20 SDs if the total amount of careless responses exceeded 6.25%, 12.5%, and 12.5% for easy, moderately difficult, and difficult tests, respectively, and b) listwise deleting data from unmotivated examinees artificially inflated the true mean by as much as .42 SDs when ability was related to careless responding. Findings from this study have implications for when and how practitioners should handle careless responses for group-based low-stakes assessments.
30 March 2016 in Applied Measurement in Education
Unmotivated test takers using rapid guessing in item responses can affect validity studies and teacher and institution performance evaluation negatively, making it critical to identify these test takers. The authors propose a new nonparametric method for finding response-time thresholds for flagging item responses that result from rapid-guessing behavior. Using data from a low-stakes assessment of college-level academic skills as an illustration, the authors evaluate and compare model fit and score estimation based on data sets cleaned by both new and existing methods. Flagging rapid-guessing responses is found to generally improve model fit, item parameter estimation, and score estimation, as in the literature. This new method, based on both response time and response accuracy, shows promise in detecting rapid guessing and in improving efficiency of the flagging process when built into data analysis.
3 April 2015 in Educational Assessment
18 March 2013 in Educational Research and Evaluation
There are numerous statistical procedures for detecting items that function differently across subgroups of examinees that take a test or survey. However, in endeavouring to detect items that may function differentially, selection of the statistical method is only one of many important decisions. In this article, we discuss the important decisions that affect investigations of differential item functioning (DIF) such as choice of method, sample size, effect size criteria, conditioning variable, purification, DIF amplification, DIF cancellation, and research designs for evaluating DIF. Our review highlights the necessity of matching the DIF procedure to the nature of the data analysed, the need to include effect size criteria, the need to consider the direction and balance of items flagged for DIF, and the need to use replication to reduce Type I errors whenever possible. Directions for future research and practice in using DIF to enhance the validity of test scores are provided.