Decades of scientific research have been conducted on developing and evaluating methods for automated emotion recognition. With exponentially growing technology, there is a wide range of emerging applications that require emotional state recognition of the user. This paper investigates a robust approach for multimodal emotion recognition during a conversation. Three separate models for audio, video and text modalities are structured and fine-tuned on the MELD. In this paper, a transformer-based crossmodality fusion with the EmbraceNet architecture is employed to estimate the emotion. The proposed multimodal network architecture can achieve up to 65% accuracy, which significantly surpasses any of the unimodal models. We provide multiple evaluation techniques applied to our work to show that our model is robust and can even outperform the state-of-the-art models on the MELD.

Untethered mobile microrobots controlled by an external magnetic gradient field can be employed as advanced biomedical applications inside the human body such as cell therapy, micromanipulation, and noninvasive surgery. Haptic technology and telecommunication, on the other hand, can extend the potentials of untethered microrobot applications. In those applications, users can communicate with the robot operating system remotely to manipulate microrobots with haptic feedback. Haptic sensations artificially constructed by the wirelessly communicated information can assist human operators to experience forces while controlling the microrobots. The proposed system is composed of a haptic device and a magnetic tweezer system, both of which are integrated through a teleoperation technique based on network communication. Users can control the microrobots remotely and feel the haptic interactions with the remote environment in real-time. The 3D haptic environment is reconstructed dynamically by a model-free haptic rendering algorithm using a 2D planar image input of the microscope. The interaction between microrobots and environmental objects is haptically rendered as 3D objects to achieve spatial haptic operation with obstacle avoidance. Moreover, path generation and path guidance forces provide virtual interaction for human users to manipulate the microrobot by following the near-optimal path in path-following tasks. The potential applications of the presented system are medical remote treatment in different sites, remote drug delivery by avoiding physically penetrating through the skin, remotely-controlled cell manipulations, and biopsy without a biopsy needle.

The robotics field represents the integration of multiple facets of computer science and engineering. Robotics-based activities have been shown to encourage K-12 students to consider careers in computing and have even been adopted as part of core computer-science curriculum at a number of universities. Unfortunately, for students with visual impairments, there are still inadequate opportunities made available for teaching basic computing concepts using robotics-based curriculum. This outcome is generally due to the scarcity of accessible interfaces to educational robots and the unfamiliarity of teachers with alternative (e.g., nonvisual) teaching methods. As such, in this paper, we discuss the use of alternative interface modalities to engage students with visual impairments in robotics-based programming activities. We provide an overview of the interaction system and results on a pilot study that engaged nine middle school students with visual impairments during a two-week summer camp.

Social engagement is a key indicator of an individual's socio-emotional and cognitive states. For a child with Autism Spectrum Disorder (ASD), this serves as an important factor in assessing the quality of the interactions and interventions. So far, qualitative measures of social engagement have been used extensively in research and in practice, but a reliable, objective, and quantitative measure is yet to be widely accepted and utilized. In this paper, we present our work on the development of a framework for the automated measurement of social engagement in children with ASD that can be utilized in real-world settings for the long-term clinical monitoring of a child's social behaviors as well as for the evaluation of the intervention methods being used. We present a computational modeling approach to derive the social engagement metric based on a user study with children between the ages of 4 and 12 years. The study was conducted within a child-robot interaction setting that targets sensory processing skills in children. We collected video, audio and motion-tracking data from the subjects and used them to generate personalized models of social engagement by training a multi-channel and multi-layer convolutional neural network. We then evaluated the performance of this network by comparing it with traditional classifiers and assessed its limitations, followed by discussions on the next steps toward finding a comprehensive and accurate metric for social engagement in ASD.

Recently, efforts in the development of speech recognition systems and robots have come to fruition with an overflow of applications in our daily lives. However, we are still far from achieving natural interaction between humans and robots, given that robots do not take into account the emotional state of speakers. The objective of this research is to create an automatic emotion classifier integrated with a robot, such that the robot can understand the emotional state of a human user by analyzing the speech signals from the user. This becomes particularly relevant in the realm of using assistive robotics to tailor therapeutic techniques towards assisting children with autism spectrum disorder (ASD). Over the past two decades, the number of children being diagnosed with ASD has been rapidly increasing, yet the clinical and societal support have not been enough to cope with the needs. Therefore, finding alternative, affordable, and accessible means of therapy and assistance has become more of a concern. Improving audio-based emotion prediction for children with ASD will allow for the robotic system to properly assess the engagement level of the child and modify its responses to maximize the quality of interaction between the robot and the child and sustain an interactive learning environment.

Based on years of research establishing the utility of socially assistive robots (SARs) for autism spectrum disorder (ASD) intervention, such robots have now become popular tools, widely used in special education schools, autism care centers, and clinical settings. Most previous studies have explored the roles of SARs as instructors, learning aides, and social-skills trainers, focusing on the learning, language, and social impairments associated with ASD. This article addresses aspects of empathy and emotion regulation (ER) impairments, which are important underlying factors for many atypicalities manifested in ASD. We discuss the design of our robot's emotional capabilities, its emotion-based action library, and the algorithm it uses to regulate a user?s emotions. In addition, we describe a user study that evaluates the ER capabilities of an emotionally expressive empathetic agent as well as its capability to prime higher social engagement in a user.