Jing Qian

Jing Qian

Auto-generated profile, based on publicly available information

Jing Qian

New York University, New York, New York, USA

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 November 2023 in IEEE Transactions on Visualization and Computer Graphics
Verified
Journal Article
Closed access
SCSonia Castelo
Sonia Castelo
JRJoao Rulff
Joao Rulff
EMErin McGowan
Erin McGowan
BSBea Steers
Bea Steers
GWGuande Wu
Guande Wu
CSClaudio Silva
Claudio Silva
The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi-year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine-tune the components of the AR assistant.

25 April 2025
Verified
Conference Paper
Open access
CLChenyi Li
Chenyi Li
GWGuande Wu
Guande Wu
GCGromit Yeuk-Yin Chan
Gromit Yeuk-Yin Chan
DTDishita Gdi Turakhia
Dishita Gdi Turakhia
SQSonia Castelo Quispe
Sonia Castelo Quispe
JQJing Qian
Jing Qian

11 May 2024
Verified
Conference Paper
Open access
GWGuande Wu
Guande Wu
JQJing Qian
Jing Qian
SQSonia Castelo Quispe
Sonia Castelo Quispe
SCShaoyu Chen
Shaoyu Chen
JRJoão Rulff
João Rulff
CSClaudio Silva
Claudio Silva
Text presented in augmented reality provides in-situ, real-time information for users. However, this content can be challenging to apprehend quickly when engaging in cognitively demanding AR tasks, especially when it is presented on a head-mounted display. We propose ARTiST, an automatic text simplification system that uses a few-shot prompt and GPT-3 models to specifically optimize the text length and semantic content for augmented reality. Developed out of a formative study that included seven users and three experts, our system combines a customized error calibration model with a few-shot prompt to integrate the syntactic, lexical, elaborative, and content simplification techniques, and generate simplified AR text for head-worn displays. Results from a 16-user empirical study showed that ARTiST lightens the cognitive load and improves performance significantly over both unmodified text and text modified via traditional methods. Our work constitutes a step towards automating the optimization of batch text data for readability and performance in augmented reality.

27 March 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Verified
Journal Article
Open access
JMJiaju Ma
Jiaju Ma
JQJing Qian
Jing Qian
TZTongyu Zhou
Tongyu Zhou
JHJeff Huang
Jeff Huang
We propose FocalPoint, a direct manipulation technique in smartphone augmented reality (AR) for selecting small densely-packed objects within reach, a fundamental yet challenging task in AR due to the required accuracy and precision. FocalPoint adaptively and continuously updates a cylindrical geometry for selection disambiguation based on the user's selection history and hand movements. This design is informed by a preliminary study which revealed that participants preferred selecting objects appearing in particular regions of the screen. We evaluate FocalPoint against a baseline direct manipulation technique in a 12-participant study with two tasks: selecting a 3 mm wide target from a pile of cubes and virtually decorating a house with LEGO pieces. FocalPoint was three times as accurate for selecting the correct object and 5.5 seconds faster on average; participants using FocalPoint decorated their houses more and were more satisfied with the result. We further demonstrate the finer control enabled by FocalPoint in example applications of robot repair, 3D modeling, and neural network visualizations.

5 July 2018 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Verified
Journal Article
Open access
JQJing Qian
Jing Qian
ACArielle Chapin
Arielle Chapin
APAlexandra Papoutsaki
Alexandra Papoutsaki
FYFumeng Yang
Fumeng Yang
KNKlaas Nelissen
Klaas Nelissen
JHJeff Huang
Jeff Huang
Remotion is an end-to-end system for capturing and replaying rich mobile device interactions, comprising both on-screen video and physical device motions. The blueprints and software provided here allow an interface to be instrumented with Remotion's capture and visualization system. Remotion is able to mimic mobile device motion through a software 3D graphical visualization and a robotic mount that replicates the movements of a mobile device from afar. Deployed together, experimenters can emulate the mobile device postures of a remote user as if they were in the room. This is important since many usability studies are carried remotely and the contribution and scale of those studies are irreplaceable. We compared how HCI experts ("analysts") observed remote users behavioral data across three replay platforms: a traditional live time-series of motion, Remotion's software visualization, and Remotion's hardware visualization. We found that Remotion can assist analysts to infer the user's attention, emotional state, habits, and active hand posture; Remotion also has a reduced effect on mental demand for analysts when analyzing the remote user's contextual information.

11 August 2023 in arXiv
Verified
Preprint
Open access
PDF file
SCSonia Castelo
Sonia Castelo
JRJoao Rulff
Joao Rulff
EMErin McGowan
Erin McGowan
BSBea Steers
Bea Steers
GWGuande Wu
Guande Wu
CSClaudio Silva
Claudio Silva
The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine tune the components of the AR assistant.