Resilience describes the capacity of systems to react to, withstand, adjust to, and recover from disruptive events. Despite numerous metrics proposed to quantify resilience, few studies predict these metrics or the restoration time to nominal performance levels, and these studies often focus on a single domain. This paper introduces three methods to model system performance and resilience metrics, which are applicable to various engineering and social science domains. These models utilize reliability engineering techniques, including bathtub-shaped functions, mixture distributions, and regression analysis incorporating event intensity covariates. Historical U.S. job loss data during recessions are used to evaluate these approaches’ predictive accuracy. This study computes goodness-of-fit measures, confidence intervals, and resilience metrics. The results show that bathtub-shaped functions and mixture distributions accurately predict curves possessing V, U, L, and J shapes but struggle with W and K shapes involving multiple disruptions or sudden performance drops. In contrast, covariate-based models effectively track all curve types, including complex W and K shapes, like the successive shocks in the 1980 U.S. recession and the sharp decline in the 2020 U.S. recession. These models achieve a high predictive accuracy for future performance and resilience metrics, evidenced by the low sum of square errors and high adjusted coefficients of determination.
Traditional Deep Neural Networks often struggle with new or unfamiliar data patterns since they operate on a closed-set assumption. This challenge arises due to inherent limitations in the model architecture, such as the softmax function commonly used for classification tasks, which tends to exhibit overconfidence and inaccuracies when faced with novel inputs. Prior studies have highlighted the need for open-set recognition (OSR) techniques to differentiate between known and unknown data points, but existing approaches often exhibit a bias toward flagging inputs as unknown. To address this issue, we introduce a novel OSR technique called VarMax, designed to maintain a balanced approach. VarMax leverages the variance in model predictions to discern between known and unknown inputs. We propose a method for classifying ambiguous samples based on prediction variance to detect out-of-distribution samples to enhance classification accuracy and reliability. Our experiments demonstrate that VarMax meets and exceeds the performance of existing methods in identifying unknown data points while also improving the model's confidence and robustness in distinguishing between known and unknown inputs.
5 November 2024 in Annual Conference of the PHM Society
This research implements Prognostics and Health Management (PHM) using multiple linear regression and multivariate time series models to monitor and predict when the performance of a Machine Learning-based cyberattack classifier might degrade to an unacceptable level, enabling preemptive maintenance strategies.
In the digital era, cloud-based video transcoding services are essential for platforms like Netflix and YouTube, relying on virtual machines for efficient processing. While prior research has primarily explored performance evaluation, limited attention has been given to the resilience of these services against emerging disruptions. This paper aims to predict the processing rate of video transcoding services to evaluate the resilience of small, medium, and large VMs within a cloud environment by applying four regression models. Findings reveal that a mixed regression model incorporating covariates, their interactions, and polynomial terms provides the most accurate prediction of performance and resilience. Among the VMs, the small VM exhibits the highest resilience, sustaining robust performance across varying conditions, with mean time to failure identified as the key factor in its sustained operation. These insights underscore the significance of predictive models for forecasting resilience, supporting proactive decisions for preventive maintenance and fault-tolerant design to reduce cloud service downtime.