Large Language Model (LLM) assistants, such as ChatGPT, have emerged as potential alternatives to search methods for helping users navigate complex, feature-rich software. LLMs use vast training data from domain-specific texts, software manuals, and code repositories to mimic human-like interactions, offering tailored assistance, including step-by-step instructions. In this work, we investigated LLM-generated software guidance through a within-subject experiment with 16 participants and follow-up interviews. We compared a baseline LLM assistant with an LLM optimized for particular software contexts, SoftAIBot, which also offered guidelines for constructing appropriate prompts. We assessed task completion, perceived accuracy, relevance, and trust. Surprisingly, although SoftAIBot outperformed the baseline LLM, our results revealed no significant difference in LLM usage and user perceptions with or without prompt guidelines and the integration of domain context. Most users struggled to understand how the prompt’s text related to the LLM’s responses and often followed the LLM’s suggestions verbatim, even if they were incorrect. This resulted in difficulties when using the LLM’s advice for software tasks, leading to low task completion rates. Our detailed analysis also revealed that users remained unaware of inaccuracies in the LLM’s responses, indicating a gap between their lack of software expertise and their ability to evaluate the LLM’s assistance. With the growing push for designing domain-specific LLM assistants, we emphasize the importance of incorporating explainable, context-aware cues into LLMs to help users understand prompt-based interactions, identify biases, and maximize the utility of LLM assistants.
A variety of consumer Augmented Reality (AR) applications have been released on mobile devices and novel immersive headsets over the last five years, creating a breadth of new AR-enabled experiences. However, these applications, particularly those designed for immersive headsets, require users to employ unfamiliar gestural input and adopt novel interaction paradigms. To better understand how everyday users discover gestures and classify the types of interaction challenges they face, we observed how 25 novices from diverse backgrounds and technical knowledge used four different AR applications requiring a range of interaction techniques. A detailed analysis of gesture interaction traces showed that users struggled to discover the correct gestures, with the majority of errors occurring when participants could not determine the correct sequence of actions to perform or could not evaluate their actions. To further reflect on the prevalence of our findings, we carried out an expert validation study with eight professional AR designers, engineers, and researchers. We discuss implications for designing discoverable gestural input techniques that align with users' mental models, inventing AR-specific onboarding and help systems, and enhancing system-level machine recognition.
19 February 2024 in IEEE Transactions on Visualization and Computer Graphics
Informal learners of computational skills often find it difficult to self-direct their learning pursuits, which may be spread across different mediums and study sessions. Inspired by self-monitoring interventions from domains such as health and productivity, we investigate key requirements for helping informal learners better self-reflect on their learning experiences. We carried out two elicitation studies with article-based and interactive probes to explore a range of manual, automatic, and semi-automatic design approaches for capturing and presenting a learner's data. We found that although automatically generated visual overviews of learning histories are initially promising for increasing awareness, learners prefer having controls to manipulate overviews through personally relevant filtering options to better reflect on their past, plan for future sessions, and communicate with others for feedback. To validate our findings and expand our understanding of designing self-monitoring tools for use in real settings, we gathered further insights from experts, who shed light on factors to consider in terms of data collection techniques, designing for reflections, and carrying out field studies. Our findings have several implications for designing learner-centered self-monitoring interventions that can be both useful and engaging for informal learners.
Software developers new to creating Augmented Reality (AR) experiences often gravitate towards simplified development environments, such as 3D game engines. While popular game engines such as Unity and Unreal have evolved to offer extensive support and functionalities for AR creation, many developers still find it difficult to realize their immersive development projects. We ran an observational study with 12 software developers to assess how they approach the initial AR creation processes using a simplified development framework, the information resources they seek, and how their learning experience compares to the more mainstream 2D development. We observed that developers often started by looking for code examples rather than breaking down complex problems, leading to challenges in visualizing the AR experience. They encountered vocabulary issues and found trial-and-error methods ineffective due to a lack of familiarity with 3D environments, physics, and motion. These observations highlight the distinct needs of emerging AR developers and suggest that conventional code reuse strategies in mainstream development may be less effective in AR. We discuss the importance of developing more intuitive training and learning methods to foster diversity in developing interactive systems and support self-taught learners.
Motion graphics authoring is a time-intensive endeavor, demanding proficiency in various feature-rich software. Automated, example-based solutions are now being explored to simplify the motion graphics creation process. To investigate how such streamlined authoring tools impact motion designers’ workflows and perceptions of creativity, we deployed an end-to-end motion graphics authoring tool to 14 users, spanning casual to professional design expertise. Our key findings reveal a dichotomy: casual designers embraced the tool’s automation, finding empowerment in its simplicity, even at the expense of losing narrative control. Conversely, professionals expressed reservations and raised concerns about the trade-offs between efficiency and creative autonomy. Notably, the level of automation in animation emerged as a point of contention, underscoring differing expectations between the two groups. Our work contributes insights into such nuances, offering implications for designing the next generation of motion graphics authoring tools that cater to a broad spectrum of creative aspirations and abilities.