Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in which the system remains in control; interactive machine learning, in which there is a closer interaction between users and learning systems; and machine teaching, where human domain experts have control over the learning process. Aside from control, humans can also be involved in the learning process in other ways. In curriculum learning human domain experts try to impose some structure on the examples presented to improve the learning; in explainable AI the focus is on the ability of the model to explain to humans why a given solution was chosen. This collaboration between AI models and humans should not be limited only to the learning process; if we go further, we can see other terms that arise such as Usable and Useful AI. In this paper we review the state of the art of the techniques involved in the new forms of relationship between humans and ML algorithms. Our contribution is not merely listing the different approaches, but to provide definitions clarifying confusing, varied and sometimes contradictory terms; to elucidate and determine the boundaries between the different methods; and to correlate all the techniques searching for the connections and influences between them.

A major obstacle to the implantation of User-Centered Design in the real world is the fact that no precise definition of the concept of usability exists that is widely accepted and applied in practice. Generally speaking, the literature tends to define usability in overly brief and ambiguous terms and to describe its application in informal terms. This is one of the main reasons why ad hoc techniques predominate in usability study methodologies. The aims of this article are to investigate the concept of usability and to describe it by means of a detailed taxonomy that is organized hierarchically and that contains exhaustive descriptions of usability attributes. This taxonomy can be used to support different stages in the development of usable systems.

Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and inexpensive. This technique is based on following a given set of heuristics, which are typically defined as broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some limitations in Nielsen’s heuristics and some inconsistencies between them and established usability models, including Nielsen’s own.

Any machine learning (ML) model is highly dependent on the data it uses for learning, and this is even more important in the case of deep learning models. The problem is a data bottleneck, i.e. the difficulty in obtaining an adequate number of cases and quality data. Another issue is improving the learning process, which can be done by actively introducing experts into the learning loop, in what is known as human-in-the-loop (HITL) ML. We describe an ML model based on a neural network in which HITL techniques were used to resolve the data bottleneck problem for the treatment of pancreatic cancer. We first augmented the dataset using synthetic cases created by a generative adversarial network. We then launched an active learning (AL) process involving human experts as oracles to label both new cases and cases by the network found to be suspect. This AL process was carried out simultaneously with an interactive ML process in which feedback was obtained from humans in order to develop better synthetic cases for each iteration of training. We discuss the challenges involved in including humans in the learning process, especially in relation to human–computer interaction, which is acquiring great importance in building ML models and can condition the success of a HITL approach. This paper also discusses the methodological approach adopted to address these challenges.

The interest in developing usable products has led to the inclusion of usability aspects in product development processes. Nonetheless, the fact that there is a tendency to overlook characteristics of the context in which a product is to be used means that the usability of a product in its operational environment is often diminished. One of the main reasons why this is the case is because there is no clear and sufficiently detailed model available for defining the concept of context of use. A comprehensive taxonomy that describes context of use and its attributes by means of precise definitions is proposed. This taxonomy will serve as a basis for improving the validity of usability activities by enabling an analysis of the conditions of use of a product in usability studies in a structured way.