Many researchers have worked to improve the tests. They needed to demonstrate that their improved tests performed better than alternative tests. Furthermore, to obtain the suitable test is another problem among several alternatives under any specific condition. In such cases, Monte-Carlo simulation studies are used to compare the performance of tests in terms of power and Type I error probability. However, any comparison of the powers is invalid when Type I error probabilities are different. Although Zhang and Boos ( 1994 Zhang, J. , and D. D. Boos. 1994. Adjusted power estimates in Monte-Carlo experiments. Communications in Statistics-Simulation and Computation 23 (1):165–73. doi:10.1080/03610919408813162. [Taylor & Francis Online], [Web of Science ®] , [Google Scholar] ) and Lloyd (2005a) proposed solutions to overcome this problem, their proposals have some deficiencies. Zhang and Boos’s adjusted power could not be used for tests when they have same test statistics and Lloyd’s intrinsic power is not useful when Type I error probability is lower than the nominal level. In this article, a new criterion is introduced, called penalized power, to compare the power of tests when Type I error probabilities are different. A Monte-Carlo simulation study is conducted to show the efficiency of penalized power over the alternatives.
26 November 2024 in Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi
In the past decade, we have experienced a massive boom in the usage of digital solutions in higher education. Due to this boom, large amounts of data have enabled advanced data analysis methods to support learners and examine learning processes. One of the dominant research directions in learning analytics is predictive modeling of learners' success using various machine learning methods. To build learners' and teachers' trust in such methods and systems, exploring the methods and methodologies that enable relevant stakeholders to deeply understand the underlying machine-learning models is necessary. In this context, counterfactual explanations from explainable machine learning tools are promising. Several counterfactual generation methods hold much promise, but the features must be actionable and causal to be effective. Thus, obtaining which counterfactual generation method suits the student success prediction models in terms of desiderata, stability, and robustness is essential. Although a few studies have been published in recent years on the use of counterfactual explanations in educational sciences, they have yet to discuss which counterfactual generation method is more suitable for this problem. This paper analyzed the effectiveness of commonly used counterfactual generation methods, such as WhatIf Counterfactual Explanations, Multi-Objective Counterfactual Explanations, and Nearest Instance Counterfactual Explanations after balancing. This contribution presents a case study using the Open University Learning Analytics dataset to demonstrate the practical usefulness of counterfactual explanations. The results illustrate the method's effectiveness and describe concrete steps that could be taken to alter the model's prediction.
The Generalized F-test is derived based on the Generalized P-value Method to test the equality of normally distributed group means under unequal variances. There are two approaches to compute the p-value of the GF test, based on beta and chi-squared random numbers. From prior art in the literature, it appears that the two computation approaches of the Generalized tests are equivalent. In this study, the equivalence of these approaches is investigated in an extensive Monte-Carlo simulation study in terms of Type I error probability and penalized power. It is found that the equivalence of the computation approaches is not quite correct and that there is a difference between their conclusion, and researchers should decide which one is powerful than the others according to the structure of data, such as sample size, and the number of groups. Also, real data examples are given to show the opposite decisions of the computation approaches.
Imbalanced data poses a significant challenge in classification as model performance is affected by insufficient learning from minority classes. Balancing methods are often used to address this problem. However, such techniques can lead to problems such as overfitting or loss of information. This study addresses a more challenging aspect of balancing methods - their impact on model behavior. To capture these changes, Explainable Artificial Intelligence tools are used to compare models trained on datasets before and after balancing. In addition to the variable importance method, this study uses the partial dependence profile and accumulated local effects techniques. Real and simulated datasets are tested, and an open-source Python package edgaro is developed to facilitate this analysis. The results obtained show significant changes in model behavior due to balancing methods, which can lead to biased models toward a balanced distribution. These findings confirm that balancing analysis should go beyond model performance comparisons to achieve higher reliability of machine learning models. Therefore, we propose a new method performance gain plot for informed data balancing strategy to make an optimal selection of balancing method by analyzing the measure of change in model behavior versus performance gain.