In the last couple of years, augmented reality (AR) has become one of the most anticipated emerging technologies. The technology is becoming more intimate with people's everyday lives, and user experiences are getting more vivid year by year. However, most of the current AR experiences rely on interacting with virtual content augmented on the real world, becoming less impressive and exciting than they were in the first place. In this article, we explore a new perspective on enriching AR experiences for their emerging mature state. We first investigate interaction schemes in AR environments in detail and highlight potential features in the context of interacting with the real world. We also provide a brief discussion on technical challenges and issues to realize interacting with the real world in online AR scenarios. We believe that this article would initiate new research interests and directions for the next level of AR experiences.

This article presents AlterVerse, a framework that provides users with novel experiences through visual illusions of interactively altering physical objects in extended reality environments. In the AlterVerse framework, physical objects are seamlessly virtualized by aligning their 3-D virtual replicas with them. It enables users to manipulate (move or rotate) and reshape (deform the shapes or transform the styles of) the physical objects virtually. Simultaneously, the physical objects are visually removed from a physical environment, creating the perception that the altering occurs directly on the physical objects. As a promising application of the AlterVerse framework, the article demonstrates interactive virtual altering of home furniture in interior design or decoration scenarios without physically altering existing furniture.

This paper proposes a framework that allows 3D freeform manipulation of a face in live video. Unlike existing approaches, the proposed framework provides natural 3D manipulation of a face without background distortion and interactive face editing by a user’s input, which leads to freeform manipulation without any limitation of range or shape. To achieve these features, a 3D morphable face model is fitted to a face region in a video frame and is deformed by the user’s input. The video frame is then mapped as a texture to the deformed model, and the model is rendered on the video frame. Because of the high computational cost, parallelization and acceleration schemes are also adopted for real-time performance. Performance evaluation and comparison results show that the proposed framework is promising for 3D face editing in live video.