Multi-fingered haptics is imperative for truly immersive virtual reality (VR) experience, as many real world tasks involve finger manipulation. One of the key lacking aspect for this is the absence of technologically and economically viable wearable haptic interfaces that can simultaneously track the finger/hand motions and display multi-degree-of-freedom (DOF) contact forces. In this paper, we propose a novel wearable cutaneous haptic interface (WCHI), which consists of: 1) finger tracking modules (FTMs) to estimate complex multi-DOF finger and hand motion; and 2) cutaneous haptic modules (CHMs) to convey three-DOF contact force at the finger-tip. By opportunistically utilizing such different types of sensors as inertial measurement units (IMUs), force sensitive resistor (FSR) sensors, and soft sensors, the WCHI can track complex anatomically-consistent multi-DOF finger motion while avoiding FTM-CHM electromagnetic interference possibly stemming from their collocation in the small form-factor interface; while also providing the direction and magnitude of three-DOF finger-tip contact force, the feedback of which can significantly enhance the precision of contact force generation against variability among users via their closed-loop control. Human subject study is performed with a virtual peg insertion task to show the importance of both the multi-DOF finger tracking and the three-DOF cutaneous haptic feedback for dexterous manipulation in virtual environment.
We propose a novel wearable haptic device that can provide kinesthetic haptic feedback for stiffness rendering of virtual objects in augmented reality (AR). Rendering stiffness of objects using haptic feedback is crucial for realistic finger-based object manipulation, yet challenging particularly in AR due to the co-presence of a real hand, haptic device, and rendered AR objects in the scenes. By adopting passive actuation with a tendon-based transmission mechanism, the proposed haptic device can generate kinesthetic feedback strong enough for immersive manipulation and prevention of inter-penetration in a small-form-factor, while maximizing the wearability and minimizing the occlusion in AR usage. A selective locking module is adopted in the device to allow for the rendering of the elasticity of objects. We perform an experimental study of two-finger grasping to verify the efficacy of the proposed haptic device for finger-based manipulation in AR. We also quantitatively compare/articulate the effects of different types of feedbacks across haptic and visual sense (i.e., kinesthetic haptic feedback, vibrotactile haptic feedback, and visuo-haptic feedback) for stiffness rendering of virtual objects in AR for the first time.
28 May 2024 in International Journal of Control, Automation, and Systems
Safe, precise teleoperation demands a third-person 3D view that reveals collision clearances and task-critical geometry in full detail. Yet most systems still rely on live camera streams that offer tunnel-vision perspectives and weak depth cues, hiding hazards and denying operators the spatial context for precise manipulation. 3D Gaussian Splatting (GS) renders photorealistic views in real time, yet safe, efficient multi-view acquisition in cluttered teleoperation remains a bottleneck. We propose Human-in-the-Loop Gaussian Splatting (HIL-GS) that delivers safe, robust, and efficient 3D scene reconstruction for challenging teleoperation environments. HIL-GS combines three modules in a tightly-coupled loop: (1) motion-aware GS reconstruction that fuses RGB-D and proprioceptive sensors for drift-free and robust mapping under aggressive motions; (2) VR-based informative display that renders the GS map with contextual overlays/feedback in real time to ensure situational awareness and reconstruction completeness; and (3) finger-based control interface to guide the robot toward informative viewpoints through safe, non-redundant motions. Through simulation and real-world experiments, we demonstrate that HIL-GS outperforms traditional approaches in reconstruction quality, usability, and efficiency.
15 December 2025 in International Journal of Human–Computer Interaction