In this paper, we introduce a novel acoustic source localization in a three-dimensional (3D) space, based on a direction estimation technique. Assuming an acoustic source at a distance from adjacent microphones, its waves spread in a planar form called a planar wavefront. In our system, the directions and steering angles between the acoustic source and the microphone array are estimated based on a planar wavefront model using a delay and sum beamforming (DSBF) system and an array of two-dimensional (2D) microelectromechanical system (MEMS) microphones. The proposed system is designed with parallel processing hardware for real-time performance and implemented using a cost-effective field programmable gate array (FPGA) and a micro control unit (MCU). As shown in the experimental results, the localization errors of the proposed system were less than 3 cm when an impulsive acoustic source was generated over 1 m away from the microphone array, which is comparable to a position-based system with reduced computational complexity. On the basis of the high accuracy and real-time performance of localizing an impulsive acoustic source, such as striking a ball, the proposed system can be applied to screen-based sports simulation.

 We present an interactive virtual-reality (VR) fire extinguisher that provides both realistic viewing using a head-mounted display (HMD) and kinesthetic experiences using a pneumatic muscle and vibrotactile transducer. The VR fire extinguisher is designed to train people to use a fire extinguisher skillfully in real fire situations. We seamlessly integrate three technologies: VR, object motion tracking, and haptic feedback. A fire scene is immersed in the HMD, and a motion tracker is used to replicate a real designed object into the virtual environment to realize augmented reality. In addition, when the handle of the fire extinguisher is squeezed to release the extinguishing agent, the haptic device generates both vibrotactile and air flow tactile feedback signals, providing the same experience as that obtained while using a real fire extinguisher.

Recently, VR-based training applications have become popular and promising, as they can simulate real-world situations in a safe, repeatable, and cost-effective way. For immersive simulations, various input devices have been designed and proposed to increase the effectiveness of training. In this study, we developed a novel device that generates 3D hand motion data and provides haptic force feedback for VR interactions. The proposed device can track 3D hand positions using a combination of the global position estimation of ultrasonic sensors and the hand pose estimation of inertial sensors in real time. For haptic feedback, shape–memory alloy (SMA) actuators were designed to provide kinesthetic forces and an efficient power control without an overheat problem. Our device improves upon the shortcomings of existing commercial devices in tracking and haptic capabilities such that it can track global 3D positions and estimate hand poses in a VR space without using an external suit or tracker. For better flexibility in handling and feeling physical objects compared to exoskeleton-based devices, we introduced an SMA-based actuator to control haptic forces. Overall, our device was designed and implemented as a lighter and less bulky glove which provides comparable accuracy and performance in generating 3D hand motion data for a VR training application (i.e., the use of a fire extinguisher), as demonstrated in the experimental results.

Estimation of the motion of ball-shaped objects is essential for the operation of ball sport simulators. In this paper, we propose an estimation system for 3D ball motion, including speed and angle of projection, by using acoustic vector and infrared (IR) scanning sensors. Our system is comprised of three steps to estimate a ball motion: sound-based ball firing detection, sound source localization, and IR scanning for motion analysis. First, an impulsive sound classification based on the mel-frequency cepstrum and feed-forward neural network is introduced to detect the ball launch sound. An impulsive sound source localization using a 2D microelectromechanical system (MEMS) microphones and delay-and-sum beamforming is presented to estimate the firing position. The time and position of a ball in 3D space is determined from a high-speed infrared scanning method. Our experimental results demonstrate that the estimation of ball motion based on sound allows a wider activity area than similar camera-based methods. Thus, it can be practically applied to various simulations in sports such as soccer and baseball.

Compared to the study on tactile feedback gloves, the kinesthetic feedback device, which has been studied for the past several decades, has difficulties in interacting with the user owing to various problems, such as large size, low portability, and high power consumption. Herein, we present a bidirectional finger kinesthetic feedback device that can provide an immersive virtual reality experience using a shape memory alloy (SMA). The proposed device provides kinesthetic feedback without heterogeneity by integrating efficient power control of the SMA actuator, fast cooling of the SMA within one second, and high-precision motion-tracking technology. The implemented device delivers a gripping and hand spreading force of up to 10N each to the index and middle fingers.

Managing underground utility tunnels (UUT) poses challenges such as safety risks, limited accessibility, and high training costs. To address these, we developed an effective VR-based UUT management training simulator, integrating digital twins with a multi-degree-of-freedom (DoF) motion platform to provide an immersive, cybersickness-reducing experience. Advances in actuator and computer graphics technology have increased the research and development of virtual reality (VR)-based training simulators that use multi-DoF motion platforms. In this study, the proposed system provides precisely synchronized visual feedback through a head-mounted display and kinematic feedback via four-DoF motor-based actuators, realistically simulating UUT conditions, including momentum shifts such as falls and micro-swaying. To mitigate cybersickness, visual and kinematic acceleration control techniques are employed, ensuring users can comfortably and safely experience the UUT without stability concerns. We used a LiDAR sensor to model the actual UUT for realistic visualization in the VR environment and created various scenario-based content, such as facility installation, inspection, and device repair, based on the physical characteristics inside the UUT. This integrated approach enables users to effectively manage and operate UUT facilities in a safe, controlled virtual setting, bridging the gap between traditional training and real-world operations.