The increasing sophistication of fraud tactics necessitates advanced detection methods to protect financial assets and maintain system integrity. Various approaches based on artificial intelligence have been proposed to identify fraudulent activities, leveraging techniques such as machine learning and deep learning. However, class imbalance remains a significant challenge. We propose several solutions based on advanced generative modeling techniques to address the challenges posed by class imbalance in fraud detection. Class imbalance often hinders the performance of machine learning models by limiting their ability to learn from minority classes, such as fraudulent transactions. Generative models offer a promising approach to mitigate this issue by creating realistic synthetic samples, thereby enhancing the model’s ability to detect rare fraudulent cases. In this study, we introduce and evaluate multiple generative models, including Variational Autoencoders (VAEs), standard Autoencoders (AEs), Generative Adversarial Networks (GANs), and a hybrid Autoencoder–GAN model (AE-GAN). These models aim to generate synthetic fraudulent samples to balance the dataset and improve the model’s learning capacity. Our primary objective is to compare the performance of these generative models against traditional oversampling techniques, such as SMOTE and ADASYN, in the context of fraud detection. We conducted extensive experiments using a real-world credit card dataset to evaluate the effectiveness of our proposed solutions. The results, measured using the BEFS metrics, demonstrate that our generative models not only address the class imbalance problem more effectively but also outperform conventional oversampling methods in identifying fraudulent transactions.

The proliferation of new technologies and advancements in existing ones are altering our perspective of the world. So, continuous improvements are needed. A connected world filled with a vast amount of data was created as a result of the integration of these advanced technologies in the financial sector. The advantages of this connection came at the cost of more sophisticated and advanced attacks, such as fraudulent transactions. To address these illegal transactions, researchers and engineers have created and implemented various systems and models to detect fraudulent transactions; many of them produce better results than others. On the other hand, criminals change their strategies and technologies to imitate legitimate transactions. In this article, the objective is to propose an intelligent system for detecting fraudulent transactions using various deep learning architectures, including artificial neural networks (ANNs), recurrent neural networks (RNNs), and long short-term memory (LSTM). Furthermore, the Bayesian optimization algorithm is used for hyperparameter optimization. For the evaluation, a credit card fraudulent transaction dataset was used. Based on the many experiments conducted, the RNN architecture demonstrated better efficiency and yielded better results in a shorter computational time than the ANN LSTM architectures.