Xin Zeng

Xin Zeng

Auto-generated profile, based on publicly available information

Xin Zeng

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 October 2024 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
XZXin Zeng
Xin Zeng
XWXiaoyu Wang
Xiaoyu Wang
TZTengxiang Zhang
Tengxiang Zhang
CYChun Yu
Chun Yu
SZShengdong Zhao
Shengdong Zhao
YCYiqiang Chen
Yiqiang Chen
Existing gesture interfaces only work with a fixed set of gestures defined either by interface designers or by users themselves, which introduces learning or demonstration efforts that diminish their naturalness. Humans, on the other hand, understand free-form gestures by synthesizing the gesture, context, experience, and common sense. In this way, the user does not need to learn, demonstrate, or associate gestures. We introduce GestureGPT, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface. Our framework leverages multiple Large Language Model agents to manage and synthesize gesture and context information, then infers the interaction intent by associating the gesture with an interface function. More specifically, our triple-agent framework includes a Gesture Description Agent that automatically segments and formulates natural language descriptions of hand poses and movements based on hand landmark coordinates. The description is deciphered by a Gesture Inference Agent through self-reasoning and querying about the interaction context (e.g., interaction history, gaze data), which is managed by a Context Management Agent. Following iterative exchanges, the Gesture Inference Agent discerns the user’s intent by grounding it to an interactive function. We validated our framework offline under two real-world scenarios: smart home control and online video streaming. The average zero-shot Top-1/Top-5 grounding accuracies are 44.79%/83.59% for smart home tasks and 37.50%/73.44% for video streaming tasks. We also provide an extensive discussion that includes rationale for model selection, generalizability, and future research directions for a practical system etc.

19 April 2023
Verified
Conference Paper
Open access
XZXin Zeng
Xin Zeng
XWXiaoyu Wang
Xiaoyu Wang
ZGZhengtai Gou
Zhengtai Gou
YCYiqiang Chen
Yiqiang Chen
TZTengxiang Zhang
Tengxiang Zhang
Head-mounted displays (HMD) like AR glasses support powerful 3D displays and intuitive input modalities. However, there is a lack of collaboration between the HMD and other displays like PC monitors. In this paper, we propose WebJump, a software tool that analyzes HTML web pages and enables UI elements ‘jump’ from the PC monitor to the HMD. This way, contents like high-resolution images and long texts can take full advantage of the high-quality monitor display. Auxiliary contents like sidebars and menus, on the other hand, are offloaded to the HMD for easier access albeit with a lower display quality. We describe our software tool and explain in detail its implementation. We develop two applications to demonstrate WebJump’s feasibility and potential.