One of the key limitations of AI chatbots is the lack of human-like nonverbal communication. Although there are many research studies on video or audio emotion recognition for detecting human emotions, there is no research that combines video, audio, and ontology methods to develop an AI chatbot with human-like communication. Therefore, this research aims to develop an audio-video emotion recognition model and an emotion-ontology-based chatbot engine to improve human-like communication with emotion detection. This research proposed a novel model of cluster-based audiovisual emotion recognition for improving emotion detection with both video and audio signals and compared it with existing methods using video or audio signals only. Twenty-two audio features, the Mel spectrogram, and facial action units were extracted, and the last two were fed into a cluster-based independent transformer to learn long-term temporal dependencies. Our model was validated on three public audiovisual datasets: RAVDESS, SAVEE, and RML. The results demonstrated that the accuracy scores of the clustered transformer model for RAVDESS, SAVEE, and RML were 86.46%, 92.71%, and 91.67%, respectively, outperforming the existing best model with accuracy scores of 86.3%, 75%, and 60.2%, respectively. An emotion-ontology-based chatbot engine was implemented to make inquiry responses based on the detected emotion. A case study of the HKU Campusland metaverse was used as proof of concept of the emotional AI chatbot for nonverbal communication.
2 April 2026 in Proceedings of the ACM on Management of Data
We study Aggregation Queries over Nearest Neighbors (AQNN), which compute aggregates over the learned representations of the neighborhood of a designated query object. For example, a medical professional may be interested in the average heart rate of patients whose representations are similar to that of an insomnia patient. Answering AQNNs accurately and efficiently is challenging due to the high cost of generating high-quality representations (e.g., via a deep learning model trained on human expert annotations) and the different sensitivities of different aggregation functions to neighbor selection errors. We address these challenges by combining high-quality and low-cost representations to approximate the aggregate. We characterize value- and count-sensitive AQNNs and propose the Sampler with Precision-Recall in Target ( SPRinT ), a query answering framework that works in three steps: (1) sampling, (2) nearest neighbor selection, and (3) aggregation. We further establish theoretical bounds on sample sizes and aggregation errors. Extensive experiments on five datasets from three domains (medical, social media, and e-commerce) demonstrate that SPRinT achieves the lowest aggregation error with minimal computation cost in most cases compared to existing solutions. SPRinT 's performance remains stable as dataset size grows, confirming its scalability for large-scale applications requiring both accuracy and efficiency.
In Hong Kong, the number of elderly citizens will reach one-third of the population within the next decade. To mitigate this problem, timebanking has received attention in recent years. In timebanking, an NGO helper earns time credits through providing voluntary services (e.g., household duties) to elders. These time credits can be used to acquire other services. Although timebanking has shown the promise of promoting mutual care in many countries, its potential has not been fully utilized, due to the lack of IT and data support. We thus develop HINCare, a software platform that supports timebanking for multiple NGOs. Besides providing convenience to NGO supervisors, helpers, and elders, HINCare makes use of a heterogeneous information network (HIN) for recommending suitable helpers to elders. This is the first time a graph-based recommender system is used for such purposes. Currently, HINCare is used by 12 NGOs to serve more than 5000 users in Hong Kong. In this demonstration, participants can play the role of helpers and elders in the HINCare environment.