18 March 2019 in IEEE Transactions on Neural Networks and Learning Systems
As each imbalanced classification problem comes with its own set of challenges, the measure used to evaluate classifiers must be individually selected. To help researchers make this decision in an informed manner, experimental and theoretical investigations compare general properties of measures. However, existing studies do not analyze changes in measure behavior imposed by different imbalance ratios. Moreover, several characteristics of imbalanced data streams, such as the effect of dynamically changing class proportions, have not been thoroughly investigated from the perspective of different metrics. In this paper, we study measure dynamics by analyzing changes of measure values, distributions, and gradients with diverging class proportions. For this purpose, we visualize measure probability mass functions and gradients. In addition, we put forward a histogram-based normalization method that provides a unified, probabilistic interpretation of any measure over data sets with different class distributions. The results of analyzing eight popular classification measures show that the effect class proportions have on each measure is different and should be taken into account when evaluating classifiers. Apart from highlighting imbalance-related properties of each measure, our study shows a direct connection between class ratio changes and certain types of concept drift, which could be influential in designing new types of classifiers and drift detectors for imbalanced data streams.
1 December 2012 in Information Sciences
1 September 2018 in Information Sciences
20 January 2024 in Applied Soft Computing
1 June 2016 in Information Sciences
Confirmation is a useful concept for assessing the impact of the premise on the conclusion of a rule induced from data. Interpretation of probabilistic relationships between premise and conclusion of a rule led to four mathematical formulations of confirmation, called perspectives. The logical equivalence of these perspectives and the resulting general definition of confirmation underline the known qualitative aspect of the concept of confirmation. The quantitative aspect of confirmation is handled by definitions of particular confirmation measures. In this paper, we relate the qualitative and quantitative aspects by introducing a property of monotonicity of measures with respect to left- and right-hand side probabilities defining the perspectives. This new property permits consideration of confirmation measures in association with particular perspectives. We also identify several other properties that valuable confirmation measures should possess. A particular care is devoted to discussion of behavior of confirmation measures monotonic in different perspectives with respect to symmetry properties, taking also into account two new perspectives of Bayesian confirmation. We also prove that confirmation measures monotonic in the six perspectives are exhaustive in the sense that their set is closed under transformations related to symmetry properties. Finally, we verify which confirmation measures enjoy these properties.
19 June 2024 in ACM Transactions on Knowledge Discovery From Data
Society is increasingly relying on predictive models in fields like criminal justice, credit risk management, and hiring. To prevent such automated systems from discriminating against people belonging to certain groups, fairness measures have become a crucial component in socially relevant applications of machine learning. However, existing fairness measures have been designed to assess the bias between predictions for protected groups without considering the imbalance in the classes of the target variable. Current research on the potential effect of class imbalance on fairness focuses on practical applications rather than dataset-independent measure properties. In this article, we study the general properties of fairness measures for changing class and protected group proportions. For this purpose, we analyze the probability mass functions of six of the most popular group fairness measures. We also measure how the probability of achieving perfect fairness changes for varying class imbalance ratios. Moreover, we relate the dataset-independent properties of fairness measures described in this work to classifier fairness in real-life tasks. Our results show that measures such as Equal Opportunity and Positive Predictive Parity are more sensitive to changes in class imbalance than Accuracy Equality. These findings can help guide researchers and practitioners in choosing the most appropriate fairness measures for their classification problems.