24 March 2025 in Nature Communications
It is challenging for wearable obstacle avoidance devices to simultaneously meet practical demands of high reliability, rapid response, long-lasting duration, and usable design. Here we report a wearable obstacle avoidance device, comprising a set of self-developed glasses (weighing ~400 grams, including an ~80 grams battery) and a common smartphone. Specifically, the glasses collect the multi-modal data for comprehensive environmental perception, including video and depth modalities, and implement a depth-aided video compression module. This module not only adaptively compresses video data to reduce transmission delay to the smartphone, but also operates on a customized FPGA board featuring a multi float-point vector unit streaming processing architecture, thereby facilitating responsive and energy-efficient obstacle detection. Additionally, we design a cross-modal obstacle detection module on the smartphone, which ensures reliable detection and provides user-friendly auditory and tactile alerts by utilizing cross-modal learning based on modal correlations. Multiple indoor and outdoor experimental results demonstrate 100% collision avoidance rates, delay of less than 320 ms, and duration of approximately 11 hours.
1 October 2020 in IEEE Journal on Selected Areas in Communications
3 October 2022 in IEEE Journal on Selected Areas in Communications
Medical robots with audio-video-haptic streams, as indispensable devices for remote healthcare, are playing ever-increasing roles in mitigating the spread of infectious diseases. However, existing medical robots are far from precise and efficient because of the following two technical challenges, including i) how to ensure the haptic fidelity for precise manipulation, and ii) how to alleviate the impact of haptic streams on the quality of visual navigation for efficient operation. To this end, this work explores the benefits of edge-based cross-modal communications (CMCs), which take full advantage of potential correlations among different modalities’ streams, to realize high reliability and throughput. Specifically, to compensate for the reliability loss caused by wireless transmission, a semantic-aided cross-modal reconstruction framework is firstly designed at edge nodes for high haptic fidelity. Then, a user experience-driven stream scheduling strategy is developed to enhance the visual quality by fully leveraging edge computing and network slicing. In particular, different from traditionally interrupting audio/video stream transmission to prioritize haptic streams, we jointly schedule resources to different modalities’ streams via estimating haptic arrival time. Finally, as a classical case study, we independently construct a remote throat swab sampling platform based on CMCs to evaluate practical performance, and numerical results indicate the significant improvements in terms of various metrics.
17 August 2022 in IEEE Transactions on Multimedia
With the explosive growth of online multi-modal applications that typically include audio, video, and haptic signals, immersive experience (IE) improvement has been broadly regarded as one of the most important tasks. Compared with traditional quality of experience (QoE) improvement for online audio/video applications, it highlights two sequential technical challenges to be resolved: i) much more stringent demand of real-time improvement due to the incorporation of delay-sensitive haptic signals, and ii) high-dimensional instead of existing one-dimensional (i.e., network-level) paradigm for better online improvement. To get over this dilemma, this work systematically addresses the following three fundamental problems: i) which factors influence IE, ii) how to online improve IE, and iii) to what extent of the corresponding IE improvement can be achieved. To this end, we first comprehensively explore and categorize the influence factors on IE from various dimensions. Then, by combing network resource scheduling with the multi-domain collaboration of user profile, device specification, and application type, an online IE improvement strategy is proposed based on the efficient linear contextual bandit with the $L_{1}$-norm estimation. Finally, we derive the theoretical bound of IE improvement, scaling at a poly-logarithmical function of data dimension. Numerical results on the practical system also demonstrate the remarkable improvement on IE.
15 January 2022 in Journal of Environmental Chemical Engineering