Aim/Purpose: The aim of this study was to analyze various performance metrics and approaches to their classification. The main goal of the study was to develop a new typology that will help to advance knowledge of metrics and facilitate their use in machine learning regression algorithms Background: Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. A performance metric can be defined as a logical and mathematical construct designed to measure how close are the actual results from what has been expected or predicted. A vast variety of performance metrics have been described in academic literature. The most commonly mentioned metrics in research studies are Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), etc. Knowledge about metrics properties needs to be systematized to simplify the design and use of the metrics. Methodology: A qualitative study was conducted to achieve the objectives of identifying related peer-reviewed research studies, literature reviews, critical thinking and inductive reasoning. Contribution: The main contribution of this paper is in ordering knowledge of performance metrics and enhancing understanding of their structure and properties by proposing a new typology, generic primary metrics mathematical formula and a visualization chart Findings: Based on the analysis of the structure of numerous performance metrics, we proposed a framework of metrics which includes four (4) categories: primary metrics, extended metrics, composite metrics, and hybrid sets of metrics. The paper identified three (3) key components (dimensions) that determine the structure and properties of primary metrics: method of determining point distance, method of normalization, method of aggregation of point distances over a data set. For each component, implementation options have been identified. The suggested new typology has been shown to cover a total of over 40 commonly used primary metrics Recommendations for Practitioners: Presented findings can be used to facilitate teaching performance metrics to university students and expedite metrics selection and implementation processes for practitioners Recommendation for Researchers: By using the proposed typology, researchers can streamline development of new metrics with predetermined properties Impact on Society: The outcomes of this study could be used for improving evaluation results in machine learning regression, forecasting and prognostics with direct or indirect positive impacts on innovation and productivity in a societal sense Future Research: Future research is needed to examine the properties of the extended metrics, composite metrics, and hybrid sets of metrics. Empirical study of the metrics is needed using R Studio or Azure Machine Learning Studio, to find associations between the properties of primary metrics and their “numerical” behavior in a wide spectrum of data characteristics and business or research requirements

Data driven companies effectively use regression machine learning methods for making predictions in many sectors. Cloud-based Azure Machine Learning Studio (MLS) has a potential of expediting machine learning experiments by offering a convenient and powerful integrated development environment. The process of evaluating machine learning models in Azure MLS has certain limitations, e.g. small number of performance metrics and lack of functionality to evaluate custom built regression models with R language. This paper reports the results of an effort to build an Enhanced Evaluate Model (EEM) module which facilitates and accelerates Azure experiments development and evaluation. The EEM combines multiple performance metrics allowing for multisided evaluation of the regression models. EEM offers 4 times more metrics than built-in Azure Evaluate Model module. EEM metrics include: CoD, GMRAE, MAE, MAPE, MASE, MdAE, MdAPE, MdRAE, ME, MPE, MRAE, MSE, NRMSE_mm, NRMSE_sd, RAE, RMdSPE, RMSE, RMSPE, RSE, sMAPE, SMdAPE, SSE. Also, EEM supports evaluation of the R language based regression models. The operational Enhanced Evaluate Model module has been published to the web and openly available for experiments and extensions.

An international association advancing the multidisciplinary study of informing systems. Founded in 1998, the Informing Science Institute (ISI) is a global community of academics shaping the future of informing science.

Complexity in the Context of Information Systems Project Management

Return on Investment (ROI) is one of the most popular performance measurement and evaluation metrics. ROI analysis (when applied correctly) is a powerful tool in comparing solutions and making informed decisions on the acquisitions of information systems. The purpose of this study is to provide a systematic research of the accuracy of the ROI evaluations in the context of information systems implementations. Measurements theory and error analysis, specifically propagation of uncertainties methods, were used to derive analytical expressions for ROI errors. Monte Carlo simulation methodology was used to design and deliver a quantitative experiment to model costs and returns estimating errors and calculate ROI accuracies. Spreadsheet simulation (Microsoft Excel spreadsheets enhanced with Visual Basic for Applications) was used to implement Monte Carlo simulations. The main contribution of the study is that this is the first systematic effort to evaluate ROI accuracy. Analytical expressions have been derived for estimating errors of the ROI evaluations. Results of the Monte Carlo simulation will help practitioners in making informed decisions based on explicitly stated factors influencing the ROI uncertainties.

Background: Return on investment (ROI) is one of the most popular evaluation metrics. ROI analysis (when applied correctly) is a powerful tool of evaluating existing information systems and making informed decisions on the acquisitions. However, practical use of the ROI is complicated by a number of uncertainties and controversies. The article reveals some of these controversies in an engaging and thought-provocative manner. Purpose: The intent of this note is to highlight several of the ROI paradoxes in a format of an opinion or a viewpoint with a hope that drawing attention of the ROI practitioners and researchers to these issues will contribute to more transparent and responsible application of the ROI evaluation. Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Review of current practice. Findings: The article reveals three weaknesses of the ROI evaluations, which in the absence of the commonly accepted ROI standard, can make results of the ROI evaluations uncertain or questionable.