As systems grow more automatized, the human operator is all too often overlooked. Although human-robot interaction (HRI) can be quite demanding in terms of cognitive resources, the mental states (MS) of the operators are not yet taken into account by existing systems. As humans are no providential agents, this lack can lead to hazardous situations. The growing number of neurophysiology and machine learning tools now allows for efficient operators’ MS monitoring. Sending feedback on MS in a closed-loop solution is therefore at hand. Involving a consistent automated planning technique to handle such a process could be a significant asset. This perspective article was meant to provide the reader with a synthesis of the significant literature with a view to implementing systems that adapt to the operator’s MS to improve human-robot operations’ safety and performance. First of all, the need for this approach is detailed regarding remote operation, an example of HRI. Then, several MS identified as crucial for this type of HRI are defined, along with relevant electrophysiological markers. A focus is made on prime degraded MS linked to time-on-task and task demands, as well as collateral MS linked to system outputs (i.e., feedback and alarms). Lastly, the principle of symbiotic HRI is detailed and one solution is proposed to include the operator state vector into the system using a mixed-initiative decisional framework to drive such an interaction.

As is the case in several research domains, data sharing is still scarce in the field of Brain-Computer Interfaces (BCI), and particularly in that of passive BCIs—i.e., systems that enable implicit interaction or task adaptation based on a user's mental state(s) estimated from brain measures. Moreover, research in this field is currently hindered by a major challenge, which is tackling brain signal variability such as cross-session variability. Hence, with a view to develop good research practices in this field and to enable the whole community to join forces in working on cross-session estimation, we created the first passive brain-computer interface competition on cross-session workload estimation. This competition was part of the 3rd International Neuroergonomics conference. The data were electroencephalographic recordings acquired from 15 volunteers (6 females; average 25 y.o.) who performed 3 sessions—separated by 7 days—of the Multi-Attribute Task Battery-II (MATB-II) with 3 levels of difficulty per session (pseudo-randomized order). The data -training and testing sets—were made publicly available on Zenodo along with Matlab and Python toy code (https://doi.org/10.5281/zenodo.5055046). To this day, the database was downloaded more than 900 times (unique downloads of all version on the 10th of December 2021: 911). Eleven teams from 3 continents (31 participants) submitted their work. The best achieving processing pipelines included a Riemannian geometry-based method. Although better than the adjusted chance level (38% with an α at 0.05 for a 3-class classification problem), the results still remained under 60% of accuracy. These results clearly underline the real challenge that is cross-session estimation. Moreover, they confirmed once more the robustness and effectiveness of Riemannian methods for BCI. On the contrary, chance level results were obtained by one third of the methods—4 teams- based on Deep Learning. These methods have not demonstrated superior results in this contest compared to traditional methods, which may be due to severe overfitting. Yet this competition is the first step toward a joint effort to tackle BCI variability and to promote good research practices including reproducibility.

The design of human–robot interactions is a key challenge to optimize operational performance. A promising approach is to consider mixed-initiative interactions in which the tasks and authority of each human and artificial agents are dynamically defined according to their current abilities. An important issue for the implementation of mixed-initiative systems is to monitor human performance to dynamically drive task allocation between human and artificial agents (i.e., robots). We, therefore, designed an experimental scenario involving missions whereby participants had to cooperate with a robot to fight fires while facing hazards. Two levels of robot automation (manual vs. autonomous) were randomly manipulated to assess their impact on the participants’ performance across missions. Cardiac activity, eye-tracking, and participants’ actions on the user interface were collected. The participants performed differently to an extent that we could identify high and low score mission groups that also exhibited different behavioral, cardiac and ocular patterns. More specifically, our findings indicated that the higher level of automation could be beneficial to low-scoring participants but detrimental to high-scoring ones, and vice versa. In addition, inter-subject single-trial classification results showed that the studied behavioral and physiological features were relevant to predict mission performance. The highest average balanced accuracy (74%) was reached using the features extracted from all input devices. These results suggest that an adaptive HRI driving system, that would aim at maximizing performance, would be capable of analyzing such physiological and behavior markers online to further change the level of automation when it is relevant for the mission purpose.

Safety studies have identified attention as a recurring cause of incidents and accidents in air traffic control. However, little is known of the precise attentional states that lead to degraded ATC performance. Therefore, we surveyed 150 French en route air traffic controllers on the causes of and impacts on perceived cooperation, safety, and performance of seven degraded attentional states from the literature: task-related and task-unrelated mind wandering, mental overload, inattentional deafness and blindness, attentional entropy, and perseveration. Our findings indicated that task-related and task-unrelated mind wandering were the most prevalent but had the least impact on perceived safety. Conversely, inattentional blindness and attentional entropy were less reported but were considered a significant safety concern, while inattentional deafness affected cooperation. Most states were experienced in workload levels consistent with the literature. However, no other factor such as shift work was identified as a cause of these states. Overall, these findings suggest that “attention” is not a specific enough subject for ATC, as attentional issues can occur in various conditions and have different impacts. As far as safety is concerned, inattentional blindness should be the prime target for further research. Neuroergonomics in particular could help develop dynamic countermeasures to mitigate its impact.

In order to assess mental workload (MWL), two main streams have been proposed, each by two distinct scientific communities. On the one hand, the human factors engineering community has adopted the model-based approach which is based on questionnaires, task analyses and cognitive modeling. On the other hand, the neuroergonomics community promotes the use of brain imaging techniques and physiological measurements. Although researchers from both communities have been studying workload in laboratory settings for decades with success, its assessment in ecological settings is still a critical issue. Moreover, to our knowledge they have seldom collaborated to directly measure, benchmark and combine the pros of their respective approach for MWL assessment. This study aims at filling this gap by evaluating the impact of the various phases of a traffic pattern performed in a full-motion flight simulator on both model-based workload assessment and electrophysiological markers. Twenty-one subjects (8 PPL pilots, 7 professional pilots and 6 piloting students; 29.5 years old in average; 3 females) participated in this experiment. Electrophysiological measurements were recorded during the execution of a traffic pattern, and questionnaires were filled afterwards on a computer. There were two main questionnaires, a reference one that consisted of a simple report of the perceived mental workload per phase, and a subjective mental workload assessment questionnaire based on the Visual Auditory Cognitive and Psychomotor (VACP) model by McCracken & Aldrich (1984). This model, which has already been used in an aeronautical context, consists of the four resource channels indicated in its name. In our work, a more advanced method was applied using a modified VACP approach to assess the current mental resource demands in a multitasking situation (Honecker et al., 2016). In order to compute the model-based MWL, we used a video-based task activity analysis, as well as a task model. The structure of the task model was derived from a hierarchical task analysis of the traffic pattern, and its data were based on the participants’ answers. Further, the investigated physiological metrics were (1) the time-domain heart rate variability (HRV) extracted from the electrocardiogram (ECG), and (2) the power in the theta [4 8] Hz, alpha [8 12] Hz and beta [13 30] Hz frequency bands computed from the electroencephalography (EEG) recorded at 4 midline electrode sites (i.e. Fz, Cz, Pz, Oz). In addition, Pope and collaborators’ engagement ratio was computed (1995; beta / (alpha + theta)). Both types of features – HRV and EEG band power features - have been shown to be sensitive to workload, notably in an aeronautical context (Harrivel et al., 2017; Dehais et al., 2017). We expected the flight phase to influence all MWL metrics. Furthermore, we also expected specific physiological metrics to better reflect subjective mental workload than others. A significant flight phase effect was found on the subjective MWL reference, the model-based MWL estimation, the HRV, the power in the theta EEG frequency band, and on the EEG engagement ratio. These differences were most notably observed between the flight phases at the beginning of the experiment (Takeoff, Upwind and Crosswind) and the end of the experiment (Base, Final and Landing). They consisted of an increase of the reported subjective MWL reference, the estimation from the VACP model and the EEG ratio, as well as a decrease in theta power and in HRV towards the last phases. Therefore, the results revealed an increase in mental workload for the last flight phases. Moreover, a significant and positive correlation between the reference MWL and the model-based MWL was found. Also, a significant negative correlation between HRV and the EEG ratio at Oz was found, as well as a significant positive correlation between the subjective MWL reference and the EEG beta power at Cz. These results highlight the relevance of specific electrophysiological measures to assess workload in ecological settings as supported by their joint modulation with subjective reports. This study paves the way towards efficient multi-sensors and multi-methods mental workload assessments in ecological settings.

Error-related potentials (ErrPs) have been studied to evaluate wrong decisions or actions in several contexts. An ErrP is an electrical potential on the scalp generated by the perception of errors and occurs unwittingly. In human-robot collaboration (HRC), ErrP detection can be shown as feedback or trigger an action to adapt the system to the user. This contributes to the improvement of HRC, taking into account user performance. However, to our knowledge, the detection of ErrPs in HRC has not been widely explored, resulting in only a few studies. This preliminary systematic review will present work on ErrP-based interfaces related to adaptation, control, and neuroergonomics for HRC. 11 articles were included after the exclusion criteria of the review stages. The average accuracy of ErrP detection was between 54% and 87.19%. In most cases, the authors simulated the occurrence of unexpected behavior of the robot. The robot mistakes occurred randomly between 25% and 35% of the total trials. Some works focused on the robot learning process and adaptation between humans and robots. The mental model and the robot behavior policy were updated based on the decoded ErrPs during collaborative interactions. ErrPs played a critical role by accelerating the adaptation process. Control-related works have included ErrP detection/features as input inside the control loop or algorithm. The proposed framework enabled the learning algorithm to converge faster than conventional methods. Other studies assessed the influence of mental workload variability in the adaptation process, given that a high mental workload affects the cognitive processes needed to perceive errors. Thus, Er-rPs present advantages for enhancing HRC, and this review opens the way to further developments in the robotic domain. We will continue this work by elaborating strategies to take advantage of the ErrP detection to design pipelines and protocols that would be used in future works.