This study was an empirically substantive examination of mathematics achievement differences between urban and rural Taiwanese students at a cognitive attribute level. Participants were eighth-grade students who participated in the Trends in International Mathematics and Science Study of 1999. The rule-space method was applied to produce a diagnostic description of urban and rural students’ cognitive knowledge, abilities, and skills related to TIMSS mathematics items. The results indicated that students in urban schools performed better than those in rural schools on high-level mathematics contents (Algebra and Geometry) and abstract thinking skills (Proportional reasoning, Logical reasoning, Solution search, and Open-ended items). Furthermore, greater proportions of urban students were classified into the knowledge states with more mastery attributes and greater proportions of rural students occupied the knowledge states with fewer mastery attributes. Detailed discussion and suggestions are provided in the paper.
27 July 2020 in The Journal of Experimental Education
A simulation study was conducted to examine the efficacy of nine frequently-used HOV tests, including Levene’s tests with squared residuals and with absolute residuals, Brown and Forsythe (BF) test, Bootstrap BF test, O’Brien test, Z-variance test, Box-Scheffé (BS) test, Bartlett test, and Pseudo jackknife test under comprehensive simulation conditions. The results indicate that O’Brien, Levene’s with squared deviations, BS, BF, and Bootstrap BF tests had adequate Type I error control, especially for O’Brien test and Levene’s test with squared deviations. Under conditions of adequately controlling for Type I error, BF and Bootstrap BF tests had consistently higher power estimates across simulation conditions. Practical recommendations of guidelines for HOV test selection in factorial ANOVA designs are presented in the paper.
1 August 2010 in Educational Research and Evaluation
Traditional comparisons of test score means identify group differences in broad academic areas, but fail to provide substantive description of how the groups differ on the specific cognitive attributes required for success in the academic area. The rule space method (RSM) allows for group comparisons at the cognitive attribute level, which consists of the cognitive knowledge, skills, and abilities an individual can employ to solve a problem. In the current study, we extend RSM group comparison methods to include comparisons of the attribute characteristic curves (ACCs) and provide a method for estimating and plotting the ACCs using SAS. We further investigated Taiwanese mathematics performance on TIMSS-1999 by comparing cognitive attributes between students of different achievement levels as well as between male and female students. The results indicated the highest and lowest achieving students differed most on mastery probabilities for Algebra (C3), Open-ended items (S10), and Rule application in algebra (P4). Gender differences in mathematical skills were quite minimal for Taiwanese students. Detailed discussion of these findings is provided in the paper.
This study applied a three-step latent class analysis (LCA) approach to explore latent classes of learning strategy use and their moderation effects on the relationships between motivation and mathematics performance. The data of 15-year-old students from five East Asian educational systems related to Chinese culture in the Programme of International Student Assessment (PISA) in 2012 were analyzed. The findings indicated that Shanghai, Singapore, Taiwan, and Macau showed three latent classes of learning strategies, whereas Hong Kong had two latent classes. Most students in the five educational systems reported to use the control strategy, some students reported the use of combined learning strategies, and few students reported the use of memorization except for students in Shanghai. Furthermore, we found the moderation effects of learning strategy use on mathematics performance depended on the types of motivation and educational systems. This study provides insights into the advantages of a three-step LCA approach in educational research.
Background: Multiple-choice item (MCI) assessments are burdensome for instructors to develop. Artificial intelligence (AI, e.g., ChatGPT) can streamline the process without sacrificing quality. The quality of AI-generated MCIs and human experts is comparable. However, whether the quality of AI-generated MCIs is equally good across various domain- and task-specific prompts remains to be determined. Therefore, we ask whether AI can generate high-quality MCIs to assess learning outcomes from a psychology textbook chapter reading. Objective: In an exploratory study, we enlist Item Response Theory analysis and expert reviewers to assess MCIs generated by ChatGPT-4 from a psychology textbook chapter. Method: We submitted a prompt and textbook chapter to ChatGPT-4 requesting 20 MCIs. One hundred ninety undergraduate participants read the chapter before responding to the MCIs. Expert reviewers assessed the MCIs for learning outcome alignment and quality. Results: ChatGPT-4-generated MCIs were low in difficulty and high in discrimination. Expert reviewers found that nearly all items were logically sound, aligned with learning objectives, and met prevailing standards of MCI quality. Conclusion: When carefully prompted, ChatGPT-4 can rapidly generate high-quality MCIs to test comprehension of a psychology textbook chapter. However, due to the uniformly low difficulty of the items, we recommend enlisting ChatGPT-4 to write MCIs for formative, but not summative, assessments.
1 October 2024 in Applied Measurement in Education