Explainable Artificial Intelligence (XAI) for Autonomous Surface Vehicles (ASVs) addresses developers’ needs for model interpretation, understandability, and trust. As ASVs approach wide-scale deployment, these needs are expanded to include end user interactions in real-world contexts. Despite recent successes of technology-centered XAI for enhancing the explainability of AI techniques to expert users, these approaches do not necessarily carry over to non-expert end users. Passengers, other vessels, and remote operators will have XAI needs distinct from those of expert users targeted in a traditional technology-centered approach. We formulate a concept called ‘human-centered XAI’ to address emerging end user interaction needs for ASVs. To structure the concept, we adopt a model-based reasoning method for concept formation consisting of three processes: analogy, visualization, and mental simulation, drawing from examples of recent ASV research at the Norwegian University of Science and Technology (NTNU). The examples show how current research activities point to novel ways of addressing XAI needs for distinct end user interactions and underpin the human-centered XAI approach. Findings show how representations of (1) usability, (2) trust, and (3) safety make up the main processes in human-centered XAI. The contribution is the formation of human-centered XAI to help advance the research community’s efforts to expand the agenda of interpretability, understandability, and trust to include end user ASV interactions.

Autonomous ferries are providing new opportunities for urban transport mobility. With this change comes a new risk picture, which is characterised to a large extent by the safe transition from autonomous mode to manual model in critical situations. The paper presents a case study of applying an adapted risk assessment method based on the Scenario Analysis in the Crisis Intervention and Operability study (CRIOP) framework. The paper focuses on the applicability of the Scenario Analysis to address the human-automation interaction. This is done by presenting a case study applying the method on a prototype of a Human–Machine Interface (HMI) in the land-based control centre for an autonomous ferry. Hence, the paper presents findings on two levels: a method study and a case study. A concept of operation (CONOPS) and a preliminary hazard analysis lay the foundation for the scenario development, the analysis, and the discussion in a case study workshop. The case study involved a Scenario Analysis of a handover situation where the autonomous system asked for assistance from the operator in a land-based control centre. The results include a list of identified safety issues such as missing procedures, an alarm philosophy and an emergency preparedness plan, and a need for explainable AI. Findings from the study show that the Scenario Analysis method can be a valuable tool to address the human element in risk assessment by focusing on the operators’ ability to handle critical situations.

Autonomous urban passenger ferries have the potential to enhance urban mobility. However, despite advances in recent years, operating autonomous surface vehicles (ASVs) in urban waterways remain challenging, not least because transporting passengers raises safety standards to the fore. This paper presents the autonomous urban passenger ferry “milliAmpere2,” developed by the Norwegian University of Science and Technology (NTNU) in Trondheim, Norway. Design features and test results are presented, covering the investigation of five research questions related to human-centered design, batteries and propulsion, autonomous navigation and control, remote monitoring and control, and risk assessment. A three-week public trial of the “milliAmpere2,” held in 2022, helped to synthesize research outcomes in the context of its operational environment in a confined and well-trafficked urban waterway. The “milliAmpere2” project adds to a growing number of use cases demonstrating the viability of ASVs for passenger transportation. Outstanding challenges for future research are identified, including human-autonomy teaming, remote management of fleet operation, and interactions with passengers, traffic, and remote operators.