The demand for kiosk systems with embodied conversational agents has increased with the development of artificial intelligence. There have been attempts to utilize non-verbal cues, particularly virtual human (VH) eye contact, to enable human-like interaction. Eye contact with VHs can affect satisfaction with the system and the perception of VHs. However, when rendered in 2D kiosks, the gaze direction of a VH can be incorrectly perceived, due to a lack of stereo cues. A user study was conducted to examine the effects of the gaze behavior of VHs in multi-party conversations in a 2D display setting. The results showed that looking at actual speakers affects the perceived interpersonal skills, social presence, attention, co-presence, and competence in conversations with VHs. In a second study, the gaze perception was further examined with consideration of the Mona Lisa effect, which can lead users to believe that a VH rendered on a 2D display is gazing at them, regardless of the actual direction, within a narrow range. We also proposed the camera rotation angle fine tuning (CRAFT) method to enhance the users’ perceptual accuracy regarding the direction of the VH’s gaze.The results showed that the perceptual accuracy for the VH gaze decreased in a narrow range and that CRAFT could increase the perceptual accuracy.

The adoption of kiosks in public spaces is steadily increasing, with a trend toward providing more natural user experiences through embodied conversational agents (ECAs). To achieve human-like interactions, ECAs should be able to appropriately gaze at the speaker. However, kiosks in public spaces often face challenges, such as ambient noise and overlapping speech from multiple people, making it difficult to accurately identify the speaker and direct the ECA’s gaze accordingly. In this paper, we propose a lightweight gaze control system that is designed to operate effectively within the resource constraints of kiosks and the noisy conditions common in public spaces. We first developed a speaker detection model that identifies the active speaker in challenging noise conditions using only a single camera and microphone. The proposed model achieved a 91.6% mean Average Precision (mAP) in active speaker detection and a 0.6% improvement over the state-of-the-art lightweight model (Light ASD) (as evaluated on the noise-augmented AVA-Speaker Detection dataset), while maintaining real-time performance. Building on this, we developed a gaze control system for ECAs that detects the dominant speaker in a group and directs the ECA’s gaze toward them using an algorithm inspired by real human turn-taking behavior. To evaluate the system’s performance, we conducted a user study with 30 participants, comparing the system to a baseline condition (i.e., a fixed forward gaze) and a human-controlled gaze. The results showed statistically significant improvements in social/co-presence and gaze naturalness compared to the baseline, with no significant difference between the system and human-controlled gazes. This suggests that our system achieves a level of social presence and gaze naturalness comparable to a human-controlled gaze. The participants’ feedback, which indicated no clear distinction between human- and model-controlled conditions, further supports the effectiveness of our approach.