Deep neural networks have recently evolved into a powerful AI tool, reaching near-human performance level in many tasks, and in some tasks even surpassing it. However, a significant drawback of neural networks is the lack of explainability and interpretability — it is hard to say why a neural network arrived to a certain conclusion. This significantly limits application of neural networks in critical tasks and undermines trust in human-AI collaboration. It has been recently shown that internal representations constructed by a neural network can often be aligned with a domain ontology. This opens a promising way to provide explanations of a neural network in human terms. In this paper, we discuss the results of the experiment aimed at understanding what layers of a neural network are the most perspective for the alignment with given ontology concept. To do so, we build concept localization maps for XTRAINS — a synthetic dataset consisting of images and their ontological annotations. The importance of such maps is that they can be used for the development of efficient concept alignment heuristics. The experiment mostly supports the intuition that high-level concepts are localized mostly in the activations of last layers of a neural network (near its head), while lower-level concepts might be better extracted from middle layers.
An important drawback of deep neural networks limiting their application in critical tasks is the lack of explainability. Recently, several methods have been proposed to explain and interpret the results obtained by deep neural networks, however, the majority of these methods are targeted mostly at AI experts. Ontology-based explanation techniques seem promising, as they can be used to form explanations using domain terms (corresponding to ontology concepts) and logical statements, which is more understandable by domain experts. Recently, it has been shown, that inner representations (layer activations) of deep neural network can often be aligned with ontology concepts. However, not every concept can be matched with the output of every layer, and it can be computationally hard to identify the particular layer that can be easily aligned with the given concept, which is aggravated by the number of concepts in a typical ontology. The paper proposes an algorithm to address this problem. For each ontology concept it helps to identify neural network layer, which produces output that can be best aligned with the given concept. These connections can then be used to identify all the ontology concepts relevant to the sample and explain the network output in a user-friendly way.
Artificial intelligence (AI) systems, based on neural networks are becoming ubiquitous. However, in many cases, their application is constrained by the lack of interpretability. One of the ways of overcoming this limitation is neuro-symbolic approaches, where the neural network is complemented by some symbolic structures, expressing existing knowledge and making it understandable to humans. Such approaches can make the predictions more understandable and interpretable (due to the connection to human-understandable symbolic structures), but they can also have an additional advantage by allowing to train models with less data (due to leveraging prior knowledge). This paper focuses on one kind of neuro-symbolic approaches, where the role of symbolic structures is played by domain ontologies. The paper discusses the existing methods to build ontology-aware explainable neural networks and ways to leverage ontologies in forming the explanation. Based on the analysis, it proposes a computational framework for building ontology-aware self-explaining neural networks. The proposed framework allows several specializations, and it is shown that these specializations allow one to improve the prediction quality. Finally, the paper presents the results of a user study, showcasing that ontology-based explanations can improve the understandability of an AI model and the efficiency of human-AI interaction.
With the proliferation of deep artificial neural networks, techniques allowing end users to understand why a network came to a certain conclusion are becoming increasingly important. The lack of such understanding is becoming a limiting factor in applying deep neural networks in critical tasks, where the price of error is high. Recently it has been shown that internal representations built by a deep neural network can sometimes be aligned with concepts of a domain ontology, related to the network target. This opens an opportunity of explaining the results of a deep neural network in human terms (defined in the ontology). The paper presents the results of several experiments aimed at understanding what layers of a neural network are most perspective for the alignment with given ontology concept (characterized by its relations with the network target). The experiments were performed with several datasets (XTRAINS, SCDB) and several network architectures (including custom convolutional neural network architecture, ResNet, MobileNetV2). For these dataset-neural architecture pairs we built "concept localization maps" showing how informative is the output of each layer for predicting that given sample corresponds to a certain concept. The results of the experiments show that the concepts that are "closer" to the target concept (definition-wise) are typically better expressed (or, localized) in the last layers. Besides, the concept expression typically follows a roughly unimodal shape. We believe that these results can be used for building effective algorithms for concept extraction and improve the ontology-based explanation techniques for deep neural networks.
In many practical scenarios, decision-making by an AI model alone is undesirable or even impossible, and the use of an AI model is only part of a complex decision-making process that includes a human expert. Nevertheless, this fact is often overlooked when creating and training AI models – the model is trained to make decisions independently, which is not always optimal. The paper presents a review of methods that allow taking into account the joint work of AI and a human expert in the process of designing (in particular, training) AI systems, which more accurately corresponds to the practical application of the model, allows to increase the accuracy of decisions made by the system “human – AI model”, as well as to explicitly control other important parameters of the system (e.g., human workload). The review includes an analysis of the current literature on a given topic in the following main areas: 1) scenarios of interaction between a human and an AI model and formal problem statements for improving the efficiency of the “human – AI model” system; 2) methods for ensuring the efficient operation of the “human – AI model” system; 3) ways to assess the quality of human-model AI collaboration. Conclusions are drawn regarding the advantages, disadvantages, and conditions of applicability of the methods, as well as the main problems of existing approaches are identified. The review can be useful for a wide range of researchers and specialists involved in the application of AI for decision support.