Supervisory control environments, such as the NASA control room can induce high workload levels in situations where a single error is capable of costing millions of dollars. An intelligent system can improve human supervisor performance by monitoring the human’s workload levels and intelligently adapting the system capabilities, such as adapting the interaction medium or reallocating roles and responsibilities between the human and the system. Systems capable of responding promptly and accurately to the human’s changes in workload require a workload assessment algorithm that can detect changes to all components of workload in real time. A review of 24 workload assessment algorithms across six task domains is provided. Each algorithm is reviewed based on four criteria: sensitivity, diagnosticity, suitability, and generalizability. The majority of the reviewed algorithms were developed for a specific task domain and are unable to generalize different tasks. Further, the majority of the algorithms do not account for individual differences, only assess one or two workload components, and do not classify underload.

Humans commanding and monitoring robots’ actions are used in various high-stress environments, such as the Predator or MQ-9 Reaper remotely piloted unmanned aerial vehicles. The presence of stress and potential costly mistakes in these environments places considerable demands and workload on the human supervisors, which can reduce task performance. Performance may be augmented by implementing an adaptive workload human–machine teaming system that is capable of adjusting based on a human’s workload state. Such a teaming system requires a human workload assessment algorithm capable of estimating workload along multiple dimensions. A multi-dimensional algorithm that estimates workload in a supervisory environment is presented. The algorithm performs well in emulated real-world environments and generalizes across similar workload conditions and populations. This algorithm is a critical component for developing an adaptive human–robot teaming system that can adapt its interactions and intelligently (re-)allocate tasks in dynamic domains.

Advances in robot teleoperation have enabled groundbreaking innovations in many fields, such as space exploration, healthcare, and disaster relief. The human operator's performance plays a key role in the success of any teleoperation task, with prior evidence suggesting that operator stress and workload can impact task performance. As robot teleoperation is currently deployed in safety-critical domains, it is essential to analyze how different stress and workload levels impact the operator. We are unaware of any prior work investigating how both stress and workload impact teleoperation performance. We conducted a novel study ( $n=24$ ) to jointly manipulate users' stress and workload and analyze the user's performance through objective and subjective measures. Our results indicate that, as stress increased, over 70% of our participants performed better up to a moderate level of stress; yet, the majority of participants performed worse as the workload increased. Importantly, our experimental design elucidated that stress and workload have related yet distinct impacts on task performance, with workload mediating the effects of distress on performance ( $p< .05$ ).

Human-machine teams are deployed in a diverse range of task environments and paradigms that may have high failure costs (e.g., nuclear power plants). It is critical that the machine team member can interact with the human effectively without reducing task performance. These interactions may be used to manage the human's workload state intelligently, as the overall workload is related to task performance. Intelligent human-machine teaming systems rely on a facet of the human's state to determine how interaction occurs, but typically only consider the human's state at the current time step. Future task performance predictions may be leveraged to determine if adaptations need to occur in order to prevent future performance degradation. An individualized task performance prediction algorithm that relies on a multi-faceted human workload estimate is shown to predict a supervisor's task performance accurately. The analysis varies the prediction time frame (from 0 to 300 s) and compares results to a generalized algorithm.

The integration of human–robot teams is increasingly essential in dynamic task environments, particularly in sectors like warehouse management, assembly lines, search and rescue operations, material handling, and autonomous driving. This trend leverages the complementary strengths of humans and robots to enhance efficiency and tackle complex objectives. However, significant challenges arise due to differences in task execution, communication modes, empathy, mental model understanding, and adaptability between humans and robots. This survey article examines the complexities of human–robot collaboration (HRC), focusing on the “3Cs” of teamwork: collaboration, communication, and cognition. It introduces a novel 3Cs rating system to evaluate HRC systems, offering a comprehensive analysis of current research trends and identifying key challenges. The findings highlight a prevalent lack of robot adaptation based on human states and performance, underscoring the need for improved communication metrics and consistent definitions of collaborative frameworks. Key contributions include the development of the 3Cs rating system, an in-depth analysis of HRC research trends, and the identification of critical areas requiring further investigation to realize the full potential of human–robot teams. This article aims to guide future research and development, promoting more effective human–robot collaborations.

Multimodal fusion networks play a pivotal role in leveraging diverse sources of information for enhanced machine learning applications in aerial imagery. However, current approaches often suffer from a bias towards certain modalities, diminishing the potential benefits of multimodal data. This paper addresses this issue by proposing a novel modality utilization-based training method for multimodal fusion networks. The method aims to guide the network’s utilization on its input modalities, ensuring a balanced integration of complementary information streams, effectively mitigating the overutilization of dominant modalities. The method is validated on multimodal aerial imagery classification and image segmentation tasks, effectively maintaining modality utilization within $\pm 10\%$ of the user-defined target utilization and demonstrating the versatility and efficacy of the proposed method across various applications. Furthermore, the study explores the robustness of the fusion networks against noise in input modalities, a crucial aspect in real-world scenarios. The method showcases better noise robustness by maintaining performance amidst environmental changes affecting different aerial imagery sensing modalities. The network trained with 75.0% EO utilization achieves significantly better accuracy (81.4%) in noisy conditions (noise variance = 0.12) compared to traditional training methods with 99.59% EO utilization (73.7%). Additionally, it maintains an average accuracy of 85.0% across different noise levels, outperforming the traditional method’s average accuracy of 81.9%. Overall, the proposed approach presents a significant step towards harnessing the full potential of multimodal data fusion in diverse machine learning applications such as robotics, healthcare, satellite imagery, and defense applications.