A significant amount of work is invested in human-machine teaming (HMT) across multiple fields. Accurately and effectively measuring system performance of an HMT is crucial for moving the design of these systems forward. Metrics are the enabling tools to devise a benchmark in any system and serve as an evaluation platform for assessing the performance, along with the verification and validation, of a system. Currently, there is no agreed-upon set of benchmark metrics for developing HMT systems. Therefore, identification and classification of common metrics are imperative to create a benchmark in the HMT field. The key focus of this review is to conduct a detailed survey aimed at identification of metrics employed in different segments of HMT and to determine the common metrics that can be used in the future to benchmark HMTs. We have organized this review as follows: identification of metrics used in HMTs until now, and classification based on functionality and measuring techniques. Additionally, we have also attempted to analyze all the identified metrics in detail while classifying them as theoretical, applied, real-time, non–real-time, measurable, and observable metrics. We conclude this review with a detailed analysis of the identified common metrics along with their usage to benchmark HMTs.
In this paper, we present a novel pipelined near real-time speaker recognition architecture that enhances the performance of speaker recognition by exploiting the advantages of hybrid feature extraction techniques that contain the features of Gabor Filter (GF), Convolution Neural Networks (CNN), and statistical parameters as a single matrix set. This architecture has been developed to enable secure access to a voice-based user interface (UI) by enabling speaker-based authentication and integration with an existing Natural Language Processing (NLP) system. Gaining secure access to existing NLP systems also served as motivation. Initially, we identify challenges related to real-time speaker recognition and highlight the recent research in the field. Further, we analyze the functional requirements of a speaker recognition system and introduce the mechanisms that can address these requirements through our novel architecture. Subsequently, the paper discusses the effect of different techniques such as CNN, GF, and statistical parameters in feature extraction. For the classification, standard classifiers such as Support Vector Machine (SVM), Random Forest (RF) and Deep Neural Network (DNN) are investigated. To verify the validity and effectiveness of the proposed architecture, we compared different parameters including accuracy, sensitivity, and specificity with the standard AlexNet architecture.
18 June 2018 in International Journal of Distributed Sensor Networks
The ability of an autonomous unmanned aerial vehicle to navigate and fly precisely determines its utility and performance. The current navigation systems are highly dependent on the global positioning system and are prone to error because of global positioning system signal outages. However, advancements in onboard processing have enabled inertial navigation algorithms to perform well during short global positioning system outages. In this article, we propose an intelligent optical flow–based algorithm combined with Kalman filters to provide the navigation capability during global positioning system outages and global positioning system–denied environments. Traditional optical flow measurement uses block matching for motion vector calculation that makes the measurement task computationally expensive and slow. We propose the application of an artificial bee colony–based block matching technique for faster optical flow measurements. To effectively fuse optical flow data with inertial sensors output, we employ a modified form of extended Kalman filter. The modifications make the filter less noisy by utilizing the redundancy of sensors. We have achieved an accuracy of ~95% for all non-global positioning system navigation during our simulation studies. Our real-world experiments are in agreement with the simulation studies when effects of wind are taken into consideration.
19 March 2018 in International Journal of Artificial Intelligence in Education
As part of a perennial project, our team is actively engaged in developing new synthetic assistant (SA) technologies to assist in training combat medics and medical first responders. It is critical that medical first responders are well trained to deal with emergencies more effectively. This would require real-time monitoring and feedback for each trainee. Therefore, we introduced a voice-based SA to augment the training process of medical first responders and enhance their performance in the field. The potential benefits of SAs include a reduction in training costs and enhanced monitoring mechanisms. Despite the increased usage of voice-based personal assistants (PAs) in day-to-day life, the associated effects are commonly neglected for a study of human factors. Therefore, this paper focuses on performance analysis of the developed voice-based SA in emergency care provider training for a selected emergency treatment scenario. The research discussed in this paper follows design science in developing proposed technology; at length, we discussed architecture and development and presented working results of voice-based SA. The empirical testing was conducted on two groups as user studies using statistical analysis tools, one trained with conventional methods and the other with the help of SA. The statistical results demonstrated the amplification in training efficacy and performance of medical responders powered by SA. Furthermore, the paper also discusses the accuracy and time of task execution (t) and concludes with the guidelines for resolving the identified problems.
This work presents a multi-year study conducted at the University of Toledo, aimed at improving human–machine teaming (HMT) methods and technologies. With the advent of artificial intelligence (AI) in 21st-century machines, collaboration between humans and machines has become highly complicated for real-time applications. The penetration of intelligent and synthetic assistants (IA/SA) in virtually every field has opened up a path to the area of HMT. When it comes to crucial tasks such as patient treatment/care, industrial production, and defense, the use of non-standardized HMT technologies may pose a risk to human lives and cost billions of taxpayer dollars. A thorough literature survey revealed that there are not many established standards or benchmarks for HMT. In this paper, we propose a method to design an HMT based on a generalized architecture. This design includes the development of an intelligent collaborative system and the human team. Followed by the identification of processes and metrics to test and validate the proposed model, we present a novel human-in-the-loop (HIL) simulation method. The effectiveness of this method is demonstrated using two controlled HMT scenarios: Emergency care provider (ECP) training and patient treatment by an experienced medic. Both scenarios include humans processing visual data and performing actions that represent real-world applications while responding to a Voice-Based Synthetic Assistant (VBSA) as a collaborator that keeps track of actions. The impact of various machines, humans, and HMT parameters is presented from the perspective of performance, rules, roles, and operational limitations. The proposed HIL method was found to assist in standardization studies in the pursuit of HMT benchmarking for critical applications. Finally, we present guidelines for designing and benchmarking HMTs based on the case studies’ results analysis.
This work contributes towards building a novel architecture for the real-time detection of background sounds such as ambulance, helicopter, gunshot, and explosions that usually occurs in the medical first responder environment. Further, we also ensured that the proposed architecture is compatible with the existing natural language processing (NLP) systems. In this paper, we use frequency and time domain features for feature extraction and use classifiers like support vector machine (SVM), random forest (RF), and deep neural network (DNN) for classification purpose. Moreover, we also compared the result from the aforementioned machine learning algorithms in order to provide the proposed architecture with the best-classified output. Finally, to verify the validity and effectiveness of the proposed architecture, different samples of background sounds were trained and tested. Our results conclude that proposed architecture is able to detect and identify background sounds with 98% accuracy.