Malware has recently grown exponentially in recent years and poses a serious threat to individual users, corporations, banks, and government agencies. This can be seen from the growth of Advanced Persistent Threats (APTs) that make use of advance and sophisticated malware. With the wide availability of computer-automated tools such as constructors, email flooders, and spoofers. Thus, it is now easy for users who are not technically inclined to create variations in existing malware. Researchers have developed various defense techniques in response to these threats, such as static and dynamic malware analyses. These techniques are ineffective at detecting new malware in the main memory of the computer and otherwise require considerable effort and domain-specific expertise. Moreover, recent techniques of malware detection require a long time for training and occupy a large amount of memory due to their reliance on multiple factors. In this paper, we propose a computer vision-based technique for detecting malware that resides in the main computer memory in which our technique is faster or memory efficient. It works by taking portable executables in a virtual environment to extract memory dump files from the volatile memory and transform them into a particular image format. The computer vision-based contrast-limited adaptive histogram equalization and the wavelet transform are used to improve the contrast of neighboring pixel and to reduce the entropy. We then use the support vector machine, random forest, decision tree, and XGBOOST machine learning classifiers to train the model on the transformed images with dimensions of 112 × 112 and 56 × 56. The proposed technique was able to detect and classify malware with an accuracy rate of 97.01%. Its precision, recall, and F1-score were 97.36%, 95.65%, and 96.36%, respectively. Our finding shows that our technique in preparing dataset with more efficient features to be trained by the Machine Learning classifiers has resulted in significant performance in terms of accuracy, precision, recall, F1-score, speed and memory consumption. The performance has superseded most of the existing techniques in its unique approach.

Advanced Persistent Threat is an attack campaign in which an intruder or team of intruders establishes a long-term presence on a network to mine sensitive data, which becomes more dangerous when combined with polymorphic malware. This type of malware is not only undetectable, but it also generates multiple variants of the same type of malware in the network and remains in the system’s main memory to avoid detection. Few researchers employ a visualization approach based on a computer’s memory to detect and classify various classes of malware. However, a preprocessing step of denoising the malware images was not considered, which results in an overfitting problem and prevents us from perfectly generalizing a model. In this paper, we introduce a new data engineering approach comprising two main stages: Denoising and Re-Dimensioning. The first aims at reducing or ideally removing the noise in the malware’s memory-based dump files’ transformed images. The latter further processes the cleaned image by compressing them to reduce their dimensionality. This is to avoid the overfitting issue and lower the variance, computing cost, and memory utilization. We then built our machine learning model that implements the new data engineering approach and the result shows that the performance metrics of 97.82% for accuracy, 97.66% for precision, 97.25% for recall, and 97.57% for f1-score are obtained. Our new data engineering approach and machine learning model outperform existing solutions by 0.83% accuracy, 0.30% precision, 1.67% recall, and 1.25% f1-score. In addition to that, the computational time and memory usage have also reduced significantly.