The class imbalance problem in finance fraud datasets often leads to biased prediction towards the nonfraud class, resulting in poor performance in the fraud class. This study explores the effects of utilizing the Synthetic Minority Oversampling TEchnique (SMOTE), a Generative Adversarial Network (GAN), and their combinations to address the class imbalance issue. Their effectiveness was evaluated using a Feed-forward Neural Network (FNN), Convolutional Neural Network (CNN), and their hybrid (FNN+CNN). This study found that regardless of the data generation techniques applied, the classifier’s hyperparameters can affect classification performance. The comparisons of various data generation techniques demonstrated the effectiveness of the hybrid SMOTE and GAN, including SMOTified-GAN, SMOTE+GAN, and GANified-SMOTE, compared with SMOTE and GAN. The SMOTified-GAN and the proposed GANified-SMOTE were able to perform equally well across different amounts of generated fraud samples.

Substantial financial losses due to fraud drive the need for accurate detection algorithms. However, machine learning classifiers frequently show bias towards non-fraudulent classes due to class imbalance, where fraudulent instances occur much less frequently. Current oversampling techniques, such as the Synthetic Minority Oversampling TEchnique and Generative Adversarial Networks, generate noisy samples, produce suboptimal proportions of minority classes, and neglect majority class distributions, leading to degraded classifier performance. To address these limitations, this study investigates the feasibility of reinforcement learning (RL) for selecting generated minority samples. This study proposes a general-purpose RL-based sample selection method that is agnostic to both oversampling technique and classifier, which dynamically filters the generated minority samples using classifier feedback and information from minority and majority neighborhoods. The investigation reveals technical challenges, including sparse reward and high computational cost, which must be addressed for RL to become a practical solution for minority sample selection.