While recent work explored streaming volumetric content on-demand, there is little effort on live volumetric video streaming that bears the potential of bringing more exciting applications than its on-demand counterpart. To fill this critical gap, in this paper, we propose MetaStream, which is, to the best of our knowledge, the first practical live volumetric content capture, creation, delivery, and rendering system for immersive applications such as virtual, augmented, and mixed reality. To address the key challenge of the stringent latency requirement for processing and streaming a huge amount of 3D data, MetaStream integrates several innovations into a holistic system, including dynamic camera calibration, edge-assisted object segmentation, cross-camera redundant point removal, and foveated volumetric content rendering. We implement a prototype of MetaStream using commodity devices and extensively evaluate its performance. Our results demonstrate that MetaStream achieves low-latency live volumetric video streaming at close to 30 frames per second on WiFi networks. Compared to state-of-the-art systems, MetaStream reduces end-to-end latency by up to 31.7% while improving visual quality by up to 12.5%.
10 May 2024 in IEEE Transactions on Cloud Computing
Users on edge generate deep inference requests continuously over time. Mobile/edge devices located near users can undertake the computation of inference locally for users, e.g., the embedded edge device on an autonomous vehicle. Due to limited computing resources on one mobile/edge device, it may be challenging to process the inference requests from users with high throughput. An attractive solution is to (partially) offload the computation to a remote device in the network. In this paper, we examine the existing inference execution solutions across local and remote devices and propose an adaptive scheduler, a BPS scheduler, for continuous deep inference on collaborative edge intelligence. By leveraging data parallel, neurosurgeon, reinforcement learning techniques, BPS can boost the overall inference performance by up to $8.2 \times$ over the baseline schedulers. A lightweight compressor, FF, specialized in compressing intermediate output data for neurosurgeon, is proposed and integrated into the BPS scheduler. FF exploits the operating character of convolutional layers and utilizes efficient approximation algorithms. Compared to existing compression methods, FF achieves up to 86.9% lower accuracy loss and up to 83.6% lower latency overhead.
As communications service providers ponder ways to cater to the diverse traffic requirements of mobile applications that range from the classic telephony to modern augmented reality (AR)-related use cases, the traditional quality of service (QoS)-based radio resource management (RRM) techniques for RAN slicing that are agnostic to the intrinsic workings of applications can result in a poor quality of experience (QoE) for the end-user. We argue that in addition to QoS, RAN slicing strategies should also consider QoE for efficient resource utilization. However, without comprehensively understanding the interplay between QoS, QoE and how various RRM techniques can potentially influence them, it is impossible to incorporate QoE-driven feedback for resource allocation. Consequently, in this work, we conduct a first-of-its-kind in-depth experimental campaign on an O-RAN compliant 5G cellular testbed to evaluate the performance of the QoE metrics of three varied applications---video-enabled voice calling, cloud gaming, and AR---under various RAN slice configurations. We discuss the key findings of this elaborate study, and motivate the need for a QoE-aware RRM framework for RAN slicing.
This demo proposes a novel virtual content registration system (DeepMix) for MR applications, which integrates state-of-the-art computer vision technology and allows real-time interaction between virtual contents and arbitrary real objects in the physical environment for MR devices. DeepMix effectively utilizes different sensors on MR devices to measure the dimension and spatial location of real objects in the physical environment and improves the quality of experience (QoE) of users adaptively under various situations. Compared with state-of-the-art virtual content registration methods, DeepMix is a light-weight registration system with more flexibility, higher intelligence, and stronger adaptability.
In medical procedures such as surgery and diagnosis, it is crucial to provide doctors and nurses with up-to-date patient information. In this paper, we propose Health-MR, a portable Mixed-Reality (MR) system that helps medical staff monitor patient conditions. Health-MR consists of three components: (1) Patient Identification Recognition using face detection, (2) Medical Cloud Database for patient information retrieval, and (3) Non-invasive Heart Rate Measurement via image processing and Fast Fourier Transform (FFT). Our evaluation demonstrates that Health-MR significantly reduces the time needed to query patient information and provides remote, accurate, and real-time heart rate monitoring.