With the progress of Artificial Intelligence, intelligent agents are increasingly being deployed in tasks for which ethical guidelines and moral values apply. As artificial agents do not have a legal position, humans should be held accountable if actions do not comply, implying humans need to exercise control. This is often labeled as Meaningful Human Control (MHC). In this paper, achieving MHC is addressed as a design problem, defining the collaboration between humans and agents. We propose three possible team designs (Team Design Patterns), varying in the level of autonomy on the agent’s part. The team designs include explanations given by the agent to clarify its reasoning and decision-making. The designs were implemented in a simulation of a medical triage task, to be executed by a domain expert and an artificial agent. The triage task simulates making decisions under time pressure, with too few resources available to comply with all medical guidelines all the time, hence involving moral choices. Domain experts (i.e., health care professionals) participated in the present study. One goal was to assess the ecological relevance of the simulation. Secondly, to explore the control that the human has over the agent to warrant moral compliant behavior in each proposed team design. Thirdly, to evaluate the role of agent explanations on the human’s understanding in the agent’s reasoning. Results showed that the experts overall found the task a believable simulation of what might occur in reality. Domain experts experienced control over the team’s moral compliance when consequences were quickly noticeable. When instead the consequences emerged much later, the experts experienced less control and felt less responsible. Possibly due to the experienced time pressure implemented in the task or over trust in the agent, the experts did not use explanations much during the task; when asked afterwards they however considered these to be useful. It is concluded that a team design should emphasize and support the human to develop a sense of responsibility for the agent’s behavior and for the team’s decisions. The design should include explanations that fit with the assigned team roles as well as the human cognitive state.

AI will increasingly be used to collaborate with humans on ethical tasks. This study contributes to the need for practical methods to assess whether a human–AI system is under Meaningful Human Control (MHC). We propose three measurable components of MHC: subjective-, normative-, and moral control. To empirically evaluate the qualities of the MHC measuring approach, we developed an experiment in which a human–AI team performs triage during a pandemic outbreak. Participants performed the role of physician. Moral pressure was induced by a rapid influx of patients and limited resources. Three designs of human–AI collaboration were tested as a repeated within-subjects factor: (A) agent provides information and decision advice; (B) human assigns some patients to agent for triage; (C) human instructs agent to autonomously conduct triage on all patients. The measures were sufficiently sensitive to detect effects of the three human–AI team design on MHC: When advised by an agent (A), or when issuing tasks to an agent (B), participants felt more engaged, were able to exercise more control, and were more compliant with ethical guidelines. When the agent performed triage autonomously (C), participants reported a lower moral load, and judged the collaboration as less believable. Subjective, normative, and moral control can serve as a practical approach for assessing MHC.

Course Of Action (COA) generation is an important step in the military decision making process. In land operations it is a manual, creative process throughout which a commander and his staff take many factors into account, such as mission, terrain and weather, doctrine, threat capabilities and own capabilities. Disadvantages of this manual process are the inability of taking into account too many factors, the number of staff involved and the difficulty in evaluating COAs. New (AI-based) technologies can overcome these disadvantages and when implemented could serve as Tactical Decision Aids. This paper presents the results of research in which a step towards an automated tactical decision support tool for land operations was made. For a single SEIZE task at company level, based on doctrinal templates, COAs are selected and optimized using Genetic Algorithms based on simulated effects. An initial step in elucidating the generated COAs is undertaken by linking individual outcomes of simulation runs during the (GA) optimization process to tactical features of the terrain. ^{1 1} This paper was originally presented at the NATO Science and Technology Organization Symposium (ICMCIS) organized by the Information Systems Technology (IST) Panel, IST-205-RSY - the ICMCIS, held in Koblenz, Germany, 23–24 April 2024.