Jiannan Li

Jiannan Li

Auto-generated profile, based on publicly available information

Jiannan Li

School of Computing and Information Systems, Singapore Management University, Singapore, Singapore

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 July 2024
Verified
Conference Paper
Open access
SMSetareh Aghel Manesh
Setareh Aghel Manesh
TZTianyi Zhang
Tianyi Zhang
YOYuki Onishi
Yuki Onishi
KHKotaro Hara
Kotaro Hara
SBScott Bateman
Scott Bateman
ATAnthony Tang
Anthony Tang
Generative AI tools can provide people with the ability to create virtual environments and scenes with natural language prompts. Yet, how people will formulate such prompts is unclear—particularly when they inhabit the environment that they are designing. For instance, it is likely that a person might say, “Put a chair here,” while pointing at a location. If such linguistic and embodied features are common to people’s prompts, we need to tune models to accommodate them. In this work, we present a Wizard of Oz elicitation study with 22 participants, where we studied people’s implicit expectations when verbally prompting such programming agents to create interactive VR scenes. Our findings show when people prompted the agent, they had several implicit expectations of these agents: (1) they should have an embodied knowledge of the environment; (2) they should understand embodied prompts by users; (3) they should recall previous states of the scene and the conversation, and that (4) they should have a commonsense understanding of objects in the scene. Further, we found that participants prompted differently when they were prompting in situ (i.e. within the VR environment) versus ex situ (i.e. viewing the VR environment from the outside). To explore how these lessons could be applied, we designed and built Ostaad, a conversational programming agent that allows non-programmers to design interactive VR experiences that they inhabit. Based on these explorations, we outline new opportunities and challenges for conversational programming agents that create VR environments.

25 April 2025
Verified
Conference Paper
Open access
TZTianyi Zhang
Tianyi Zhang
CYColin Au Yeung
Colin Au Yeung
EAEmily Aurelia
Emily Aurelia
YOYuki Onishi
Yuki Onishi
NCNeil Chulpongsatorn
Neil Chulpongsatorn
ATAnthony Tang
Anthony Tang

1 November 2021
Verified
Conference Paper
Open access
ZLZhuoyue Lyu
Zhuoyue Lyu
JLJiannan Li
Jiannan Li
BWBryan Wang
Bryan Wang
Artificial Intelligence (AI), especially Neural Networks (NNs), has become increasingly popular. However, people usually treat AI as a tool, focusing on improving outcome, accuracy, and performance while paying less attention to the representation of AI itself. We present AIive, an interactive visualization of AI in Virtual Reality (VR) that brings AI “alive”. AIive enables users to manipulate the parameters of NNs with virtual hands and provides auditory feedback for the real-time values of loss, accuracy, and hyperparameters. Thus, AIive contributes an artistic and intuitive way to represent AI by integrating visualization, sonification, and direct manipulation in VR, potentially targeting a wide range of audiences.

19 April 2023
Verified
Conference Paper
Open access
JLJiannan Li
Jiannan Li
MSMauricio Sousa
Mauricio Sousa
KMKarthik Mahadevan
Karthik Mahadevan
BWBryan Wang
Bryan Wang
PAPaula Akemi Aoyagui
Paula Akemi Aoyagui
TGTovi Grossman
Tovi Grossman
Live and pre-recorded video tutorials are an effective means for teaching physical skills such as cooking or prototyping electronics. A dedicated cameraperson following an instructor’s activities can improve production quality. However, instructors who do not have access to a cameraperson’s help often have to work within the constraints of static cameras. We present Stargazer, a novel approach for assisting with tutorial content creation with a camera robot that autonomously tracks regions of interest based on instructor actions to capture dynamic shots. Instructors can adjust the camera behaviors of Stargazer with subtle cues, including gestures and speech, allowing them to fluidly integrate camera control commands into instructional activities. Our user study with six instructors, each teaching a distinct skill, showed that participants could create dynamic tutorial videos with a diverse range of subjects, camera framing, and camera angle combinations using Stargazer.

11 October 2024
Verified
Conference Paper
Open access
LSLudwig Sidenmark
Ludwig Sidenmark
TZTianyu Zhang
Tianyu Zhang
LLLeen Al Lababidi
Leen Al Lababidi
JLJiannan Li
Jiannan Li
TGTovi Grossman
Tovi Grossman
Mixed Reality enables hybrid workspaces where physical and virtual monitors are adaptively created and moved to suit the current environment and needs. However, in shared settings, individual users’ workspaces are rarely aligned and can vary significantly in the number of monitors, available physical space, and workspace layout, creating inconsistencies between workspaces which may cause confusion and reduce collaboration. We present Desk2Desk, an optimization-based approach for remote collaboration in which the hybrid workspaces of two collaborators are fully integrated to enable immersive side-by-side collaboration. The optimization adjusts each user’s workspace in layout and number of shared monitors and creates a mapping between workspaces to handle inconsistencies between workspaces due to physical constraints (e.g. physical monitors). We show in a user study how our system adaptively merges dissimilar physical workspaces to enable immersive side-by-side collaboration, and demonstrate how an optimization-based approach can effectively address dissimilar physical layouts.

16 October 2025 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
QWQian Wan
Qian Wan
JLJiannan Li
Jiannan Li
HWHuanchen Wang
Huanchen Wang
ZLZhicong Lu
Zhicong Lu
Prewriting is the process of generating and organising ideas before a first draft. It consists of a combination of informal, iterative, and semi-structured strategies such as visual diagramming, which poses a challenge for collaborating with large language models (LLMs) in a turn-taking conversational manner. We present Polymind, a visual diagramming tool that leverages multiple LLM-powered agents to support prewriting. The system features a parallel collaboration workflow in place of the turn-taking conversational interactions. It defines multiple ''microtasks'' to simulate group collaboration scenarios such as collaborative writing and group brainstorming. Instead of repetitively prompting a chatbot for various purposes, Polymind enables users to orchestrate multiple microtasks simultaneously. Users can configure and delegate customised microtasks, and manage their microtasks by specifying task requirements and toggling visibility and initiative. Our evaluation revealed that, compared to ChatGPT, users had more customizability over collaboration with Polymind, and were thus able to quickly expand personalised writing ideas during prewriting.