19 December 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Shared Mixed Reality experiences allow two co-located users to collaborate on both physical and digital tasks with familiar social protocols. However, extending the same to remote collaboration is limited by cumbersome setups for aligning distinct physical environments and the lack of access to remote physical artifacts. We present SurfShare, a general-purpose symmetric remote collaboration system with mixed-reality head-mounted displays (HMDs). Our system shares a spatially consistent physical-virtual workspace between two remote users, anchored on a physical plane in each environment (e.g., a desk or wall). The video feed of each user's physical surface is overlaid virtually on the other side, creating a shared view of the physical space. We integrate the physical and virtual workspace through virtual replication. Users can transmute physical objects to the virtual space as virtual replicas. Our system is lightweight, implemented using only the capabilities of the headset, without requiring any modifications to the environment (e.g. cameras or motion tracking hardware). We discuss the design, implementation, and interaction capabilities of our prototype, and demonstrate the utility of SurfShare through four example applications. In a user experiment with a comprehensive prototyping task, we found that SurfShare provides a physical-virtual workspace that supports low-fi prototyping with flexible proxemics and fluid collaboration dynamics.
3 October 2023 in IEEE Transactions on Visualization and Computer Graphics
Real-time communication with immersive 360° video can enable users to be telepresent within a remotely streamed environment. Increasingly, users are shifting to mobile devices and connecting to the Internet via mobile-cellular networks. As the ideal media for 360° videos, some VR headsets now also come with cellular capacity, giving them potential for mobile applications. However, streaming high-quality 360° live video poses challenges for network bandwidth, particularly on cellular connections. To reduce bandwidth requirements, videos can be compressed using viewport-adaptive streaming or foveated rendering techniques. Such approaches require very low latency in order to be effective, which has previously limited their applications on traditional cellular networks. In this work, we demonstrate an end-to-end virtual reality telepresence system that streams $\sim$6K 360° video over 5G millimeter-wave (mmW) radio. Our use of 5G technologies, in conjunction with mobile edge compute nodes, substantially reduces latency when compared with existing 4G networks, enabling high-efficiency foveated compression over modern cellular networks on par with WiFi. We performed a technical evaluation of our system's visual quality post-compression with peak signal-to-noise ratio (PSNR) and FOVVideoVDP. We also conducted a user study to evaluate users' sensitivity to compressed video. Our findings demonstrate that our system achieves visually indistinguishable video streams while using up to 80% less data when compared with un-foveated video. We demonstrate our video compression system in the context of an immersive, telepresent video calling application.
21 December 2022 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Recent advancements in object-tracking technologies can turn mundane constructive assemblies into Tangible User Interfaces (TUI) media. Users rely on instructions or their own creativity to build both permanent and temporary structures out of such objects. However, most existing object-tracking technologies focus on tracking structures as monoliths, making it impossible to infer and track the user's assembly process and the resulting structures. Technologies that can track the assembly process often rely on specially fabricated assemblies, limiting the types of objects and structures they can track. Here, we present StructureSense, a tracking system based on passive UHF-RFID sensing that infers constructive assembly structures from object motion. We illustrated StructureSense in two use cases (as guided instructions and authoring tool) on two different constructive sets (wooden lamp and Jumbo Blocks), and evaluated system performance and usability. Our results showed the feasibility of using StructureSense to track mundane constructive assembly structures.