In streaming data environments like weather forecasting, health care monitoring, network traffic monitoring and energy consumption etc., data characteristics and probability distributions are likely to change over time, posing challenges for classification models to predict accurately. In such non-stationary environments where the patterns in data can change quickly, the pre-trained machine learning models may become outdated and hence, there is a need to update the model to maintain an acceptable predictive performance. Existing approaches to drift detection, particularly supervised and unsupervised, have inherent problems. Supervised methods detect drift based on the error rate and assume that labels are available immediately after prediction which may not be possible in several real-world scenarios. Unsupervised methods on the other hand suffer from high rate of false alarms and curse of dimensionality (i.e., the complexity faced in detecting drift across an increased number of features). To develop a more applicable technology, in this paper, we are concerned with unsupervised drift detection i.e., to detect drift independent of the labels. A recent set of experiments has revealed that it is possible to make an autoencoder learn the data distribution of a pre-defined set of classes and distinguish between the different class samples as they arrive. Based on this, we propose an Autoencoder-based Drift Detection Method (AE-DDM) which monitors the distribution of reconstruction loss in an incoming batch stream based on a thresholding mechanism to generate warnings and detect drift. AE-DDM has been tested on four synthetics (RBM, Hyperplane, Stagger and Gaussian)and one real word dataset (NOAA) with sudden and gradual drifts. AE-DDM starts generating warnings and then conforms the drift with zero delays in case of sudden drift with reduced false alarms. The classification results of ten most used machine learning classifiers on the NOAA dataset show that the detected drift is real. In this context, the potential contributions of our research are: 1) a novel framework for unsupervised drift detection based on the deep learning autoencoder technology, 2) A brief comparison of available drift detection approaches with proposed properties of an ideal drift detector, c) creation of a synthetic Gaussian dataset for drift detection research, and d) use of count threshold in conjunction with a batch threshold to combat false alarms in drift detection.

In streaming data environments, data characteristics and probability distributions are likely to change over time, causing a phenomenon called concept drift, which poses challenges for machine learning models to predict accurately. In such non-stationary environments, there is a need to detect concept drift and update the model to maintain an acceptable predictive performance. Existing approaches to drift detection have inherent problems like requirements of truth labels in supervised detection methods and high false positive rate in case of unsupervised drift detection. In this paper, we propose a semi-supervised Autoencoder based Drift Detection Method (AEDDM) aimed at detecting drift without the need of truth labels, yet with a high confidence that the detected drift is real. In a binary classification setting, AEDDM uses two autoencoders in a layered architecture, trained on labelled data and uses a thresholding mechanism based on reconstruction error to signal the presence of drift. The proposed method has been evaluated on four synthetic and four real world datasets with different drifting scenarios. In case of real-world datasets, the induced and detected drifts have been evaluated from classifier’s performance viewpoint using seven mostly used batch classifiers as well as from adaptation perspective in an online learning environment using Hoeffding Tree classifier. The results show that AEDDM affectively detects the distributional changes in data which are most likely to impact the classifier’s performance (real drift) while ignoring the virtual drift thus considerably reducing the false alarms with an ability to adapt in terms of classification performance.

The occurrence of concept drift is an important phenomenon in an operational scenario related to machine learning based system which affects the performance of pre-trained models. To maintain the integrity and confidence in predictions, an effective drift detection and adaptation mechanism is a vital component in these operational systems. Apart from different supervised, semi-supervised and unsupervised drift detection techniques, recently the research trend has been focused more on deep learning-based methods specifically based on autoencoders. A hallmark of autoencoders is their versatility. Different types and implementations exist, each specializing in handling specific tasks. In case of drift detection, the focus is to learn the data distribution using autoencoder and to measure the deviation of the newly arriving data in terms of reconstruction loss. While standard or vanilla autoencoders are the most used; other variations also exist and can be evaluated for better drift detection performance. Apart from the type of the autoencoder, another important consideration is the use of right set of parameters and hyperparameters for an autoencoder based drift detection mechanism. In this paper, we provide a framework to optimize the performance of autoencoder based drift detection methods. We provide a theoretical and an empirical evaluation of other applicable types of autoencoders including denoising, variational and standard, followed by a detailed mechanism for selecting the best hyperparameters using grid search and selection of drift detection method’s specific parameters using both grid search and Bayesian Optimization (BO). For experimentation, we have used AEDDM (Autoencoder-based Drift Detection Method) as the base drift detection method to produce Opt-AEDDM- the optimized version of AEDDM. Detailed experiments on 4 synthetic (Hyperplane, Gaussian, VD and RBF) and two real world datasets (NOAA and Forest Covertype) prove the applicability and effectiveness of the proposed framework in finding the best autoencoder with best set of hyperparameters and parameters with improved drift detection performance.