The automatic emotion recognition domain brings new methods and technologies that might be used to enhance therapy of children with autism. The paper aims at the exploration of methods and tools used to recognize emotions in children. It presents a literature review study that was performed using a systematic approach and PRISMA methodology for reporting quantitative and qualitative results. Diverse observation channels and modalities are used in the analyzed studies, including facial expressions, prosody of speech, and physiological signals. Regarding representation models, the basic emotions are the most frequently recognized, especially happiness, fear, and sadness. Both single-channel and multichannel approaches are applied, with a preference for the first one. For multimodal recognition, early fusion was the most frequently applied. SVM and neural networks were the most popular for building classifiers. Qualitative analysis revealed important clues on participant group construction and the most common combinations of modalities and methods. All channels are reported to be prone to some disturbance, and as a result, information on a specific symptoms of emotions might be temporarily or permanently unavailable. The challenges of proper stimuli, labelling methods, and the creation of open datasets were also identified.

In recent years, a number of new research papers have emerged on the application of neural networks in affective computing. One of the newest trends observed is the utilization of graph neural networks (GNNs) to recognize emotions. The study presented in the paper follows this trend. Within the work, GraphSleepNet (a GNN for classifying the stages of sleep) was adjusted for emotion recognition and validated for this purpose. The key assumption of the validation was to analyze its correctness for the Circumplex model to further analyze the solution for emotion recognition in the Ekman modal. The novelty of this research is not only the utilization of a GNN network with GraphSleepNet architecture for emotion recognition, but also the analysis of the potential of emotion recognition based on differential entropy features in the Ekman model with a neutral state and a special focus on continuous emotion recognition during the performance of an activity The GNN was validated against the AMIGOS dataset. The research shows how the use of various modalities influences the correctness of the recognition of basic emotions and the neutral state. Moreover, the correctness of the recognition of basic emotions is validated for two configurations of the GNN. The results show numerous interesting observations for Ekman’s model while the accuracy of the Circumplex model is similar to the baseline methods.

The paper deals with the challenge of inconsistency in multichannel emotion recognition. The focus of the paper is to explore factors that might influence the inconsistency. The paper reports an experiment that used multi-camera facial expression analysis with multiple recognition systems. The data were analyzed using a multidimensional approach and data mining techniques. The study allowed us to explore camera location, occlusions and algorithm factors in the late fusion of emotion recognition results. We proposed to use a multidimensional data model for mining the various interdependencies between the factors of inconsistency. The study allowed the exploration of challenges in multichannel emotion recognition. It was achieved by comparing the consistency of obtained emotions and identification of rules determining conditions when the obtained emotions are consistent. However, the main novelty of the paper is the method of mining the inconsistencies. The study might be interesting both for researchers dealing with integration in emotion recognition, as well as for practitioners who use automatic emotion analysis software and expect to get valid results.

Data reusability is an important feature of current research, just in every field of science. Modern research in Affective Computing, often rely on datasets containing experiments-originated data such as biosignals, video clips, or images. Moreover, conducting experiments with a vast number of participants to build datasets for Affective Computing research is time-consuming and expensive. Therefore, it is extremely important to provide solutions allowing one to (re)use data from a variety of sources, which usually demands data integration. This paper presents the Graph Representation Integrating Signals for Emotion Recognition and Analysis (GRISERA) framework, which provides a persistent model for storing integrated signals and methods for its creation. To the best of our knowledge, this is the first approach in Affective Computing field that addresses the problem of integrating data from multiple experiments, storing it in a consistent way, and providing query patterns for data retrieval. The proposed framework is based on the standardized graph model, which is known to be highly suitable for signal processing purposes. The validation proved that data from the well-known AMIGOS dataset can be stored in the GRISERA framework and later retrieved for training deep learning models. Furthermore, the second case study proved that it is possible to integrate signals from multiple sources (AMIGOS, ASCERTAIN, and DEAP) into GRISERA and retrieve them for further statistical analysis.

One of the major challenges facing the field of Affective Computing is the reusability of datasets. Existing affective-related datasets are not consistent with each other, they store a variety of information in different forms, different formats, and the terms used to describe them are not unified. This paper proposes a new ontology, ROAD, as a solution to this problem, by formally describing the datasets and unifying the terms used. The developed ontology allows information about the origin and meaning of the data to be modeled, i.e., time series, representing both emotional states and features derived from biosignals. Furthermore, the ROAD ontology is extensible and not application-oriented, thus it can be used to store data from a wide range of Affective Computing experiments. The ontology was validated by modeling data obtained from one experiment on the AMIGOS dataset. The approach proposed in the paper can be used both by researchers who create new datasets or want to reuse existing ones, and for those who want to process data from experiments in a more automated way.

Most of the research in the field of emotion recognition is based on datasets that contain data obtained during affective computing experiments. However, each dataset is described by different metadata, stored in various structures and formats. This research can be counted among those whose aim is to provide a structural and semantic pattern for affective computing datasets, which is an important step to solve the problem of data reuse and integration in this domain. In our previous work, the ROAD ontology was introduced. This ontology was designed as a skeleton for expressing contextual data describing time series obtained in various ways from various signals and was focused on common contextual data, independent of specific signals. The aim of the presented research is to provide a carefully curated vocabulary for describing signals obtained from electrodermal activity, a very important subdomain of emotion analysis. We decided to present it as an extension to the ROAD ontology in order to offer means of sharing metadata for datasets in a unified and precise way. To meet this aim, the research methodology was defined, mostly focusing on requirements specification and integration with other existing ontologies. Application of this methodology resulted firstly in sharing the requirements to allow a broader discussion and secondly development of the EDA extension of the ROAD ontology, validated against the MAHNOB-HCI dataset. Both these results are very important with respect to the vast context of the work, i.e. providing an extendable framework for describing affective computing experiments. Introducing the methodology also opens the way for providing new extensions systematically just by executing the steps defined in the methodology.