Ensemble methods can be used to identify causal relationships in data for a better understanding and taking the right decision in processes that involve high risk. This paper explores the idea of a causal decision tree forest and proposes a regularized ensemble method by integrating optimal causal trees for improved prediction accuracy while not compromising on accurately estimating heterogeneous treatment effects. The proposed method is based on selecting a subset of the most accurate causal trees from a sufficiently large pool based on their out-of-sample error estimates. The selected trees are integrated to form an ensemble that is used for estimating heterogeneous treatment effect and predicting unseen data. The proposed method is applied on Pakistan’s income function consisting of 27964 observations on wages of workers age 10 and above as an example dataset. The paper gives a detailed simulation study where datasets are generated under 5 different designs. The proposed method is assessed against ordinary least square (OLS), least absolute shrinkage and selection operator (LASSO), Ridge, Causal Tree and the standard decision trees forest via mean square error (MSE), root mean square error (RMSE), mean absolute deviation (MAD) and Pearson correlation (r) as performance metrics. The analyses given in the paper reveal that the proposed method can be used effectively for estimating heterogeneous treatment effects and achieves better prediction performance and as compared to the rest of the methods given in the paper.

This paper presents an ensemble method for binary classification, where each base model is based on an extended neighbourhood rule (ExNRule). The ExNRule identifies the neighbours of an unseen observation in a stepwise manner. This rule first selects the sample point closest to the experimental observation and then selects the second observation nearest to the previously chosen one. To find the required data points in the neighbourhood, this search is repeated up to k steps. The test sample point is predicted using majority voting in the class labels of the k chosen neighbours. In the proposed method, a large number of ExNRule based models are constructed on randomly projected bootstrap samples. The error rates of these models are computed using out-of-bag data points. The models are then ranked according to their out-of-bag errors, and a proportion of the most accurate models are selected. The final ensemble is constructed by combining the selected models. The proposed method is compared with other classical procedures on 15 benchmark datasets in terms of classification accuracy, Kohen’s kappa and Brier score (BS) as performance metrics. Boxplots of the results are also constructed. The proposed ensemble is outperforming the existing methods on almost all the benchmark datasets. For further evaluation, the proposed method is compared with other kNN based classifiers on 3 datasets using different k values. Furthermore, the performance of the proposed method is also evaluated using simulated data under different scenarios.

This study proposes a supervised feature selection technique for classification in high dimensional binary class problems by adding robustness in the conventional Fisher Score. The proposed method utilizes the more robust measure of location, i.e. the median and measure of dispersion known as Rousseeuw and Croux statistic (Qn). Initially, a minimum subset of genes is identified by the greedy search approach, which is then combined with the top ranked genes obtained via the proposed Robust Fisher Score (RFish). To remove redundancy in the selected genes, Least Absolute Shrinkage and Selection Operator (LASSO) is then applied. The proposed method is validated on five publicly available datasets and is further assessed in a detailed simulation study. The results of the proposed method are compared with six well known feature selection methods based on prediction performance via Random Forest (RF), Support Vector Machine (SVM) and k Nearest Neighbour (k-NN) classifiers. The findings are presented in boxplots and barplots, which show that the proposed method (RFish) outperforms all the other methods in the majority of cases.

Ensembles based on k NN models are considered effective in reducing the adverse effect of outliers, primarily, by identifying the closest observations to a test point in a given training data. Class label of the test point is estimated by taking a majority vote of the nearest observations’ class labels. While identifying the closest observations, certain training patterns might possess high regulatory power than the others. Therefore, assigning weights to observations and then calculating weighted distances are deemed important. This paper proposes a k NN ensemble that identifies nearest observations based on their weighted distance in relation to the response variable via support vectors. This is done while building a large number of k NN models each on a bootstrap sample from the training data along with a randomly selected subset of features from the given feature space. The estimated class of the test observation is decided via majority voting based on the estimates given by all the base k NN models.The ensemble is assessed on 14 benchmark and simulated datasets against other classical methods, including k NN based models using Brier score, classification accuracy and Kappa as performance measures. Average values of the given measures along with boxplots are given for the proposed and the other competitors. On both benchmark and simulated datasets, the proposed ensemble outperformed the other competitive methods in majority of the cases. The suggested ensemble gave overall better classification performance than the other methods on 8 datasets. The ensemble performed better than the other methods in simulation study that considered the presence of noisy features in a dataset. The analyses reveal that the proposed method is effective in classification problems that involve noisy features in the data. Furthermore, feature weighting and randomization also make the method robust to the choice of k , i.e., the number of nearest observations in a base model.

The COVID-19 pandemic has had a significant impact on students’ academic performance. The effects of the pandemic have varied among students, but some general trends have emerged. One of the primary challenges for students during the pandemic has been the disruption of their study habits. Students getting used to online learning routines might find it even more challenging to perform well in face to face learning. Therefore, assessing various potential risk factors associated with students low performance and its prediction is important for early intervention. As students’ performance data encompass diverse behaviors, standard machine learning methods find it hard to get useful insights for beneficial practical decision making and early interventions. Therefore, this research explores regularized ensemble learning methods for effectively analyzing students’ performance data and reaching valid conclusions. To this end, three pruning strategies are implemented for the random forest method. These methods are based on out-of-bag sampling, sub-sampling and sub-bagging. The pruning strategies discard trees that are adversely affected by the unusual patterns in the students data forming forests of accurate and diverse trees. The methods are illustrated on an example data collected from university students currently studying on campus in a face-to-face modality, who studied during the COVID-19 pandemic through online learning. The suggested methods outperform all the other methods considered in this paper for predicting students at the risk of academic failure. Moreover, various factors such as class attendance, students interaction, internet connectivity, pre-requisite course(s) during the restrictions, etc., are identified as the most significant features.

This article introduces a novel method, called Graphical Group Ridge (GG-Ridge), which classifies ridge regression predictors in disjoint groups of conditionally correlated variables and derives different penalties (shrinkage parameters) for these groups of predictors. It combines the ridge regression method with the graphical model for high-dimensional data (i.e. the number of predictors, p, exceeds the number of cases, n) or ill-conditioned data (e.g. in the presence of multicollinearity among predictors). Although ridge regression is very effective with these types of data, its main shortcoming is that it applies the same penalty to all predictors, which can consequently limit the reduction in both the mean square error and the prediction mean square error, and over-shrink some predictors. This issue is addressed by the new method which reduces the mean square errors by assigning different penalties to different groups of predictors. Moreover, it reduces the extent of over-shrinking of predictors as compared to the ridge method, which is a desirable property in many fields such as finance, genetics and climate. The performance of the GG-Ridge method is investigated through two simulation studies and a real data analysis, and the results are compared with those of Ridge regression, and Elastic Net. The results indicate that the GG-Ridge outperforms these two methods in reducing the mean square errors, the prediction mean square error, and the bias of coefficients estimates.