Usability testing is vital for enhancing the user experience (UX) of interactive systems. However, analyzing test videos is complex and resource-intensive. Recent AI advancements have spurred exploration into human-AI collaboration for UX analysis, particularly through natural language. Unlike user-initiated dialogue, our study investigated the potential of proactive conversational assistants to aid UX evaluators through automatic suggestions at three distinct times: before, in sync with, and after potential usability problems. We conducted a hybrid Wizard-of-Oz study involving 24 UX evaluators, using ChatGPT to generate automatic problem suggestions and a human actor to respond to impromptu questions. While timing did not significantly impact analytic performance, suggestions appearing after potential problems were preferred, enhancing trust and efficiency. Participants found the automatic suggestions useful, but they collectively identified more than twice as many problems, underscoring the irreplaceable role of human expertise. Our findings also offer insights into future human-AI collaborative tools for UX evaluation.
AI is promising in assisting UX evaluators with analyzing usability tests, but its judgments are typically presented as non-interactive visualizations. Evaluators may have questions about test recordings, but have no way of asking them. Interactive conversational assistants provide a Q&A dynamic that may improve analysis efficiency and evaluator autonomy. To understand the full range of analysis-related questions, we conducted a Wizard-of-Oz design probe study with 20 participants who interacted with simulated AI assistants via text or voice. We found that participants asked for five categories of information: user actions, user mental model, help from the AI assistant, product and task information, and user demographics. Those who used the text assistant asked more questions, but the question lengths were similar. The text assistant was perceived as significantly more efficient, but both were rated equally in satisfaction and trust. We also provide design considerations for future conversational AI assistants for UX evaluation.
Few instructors include accessibility in computing education curriculum despite its importance in the computing field. Prior work on accessibility and CS education vary on what accessibility knowledge is appropriate for future computing professionals and is covered mainly in Human-Computer Interaction (HCI) and web design courses. In practice, there is no comprehensive list of learning objectives to characterize Accessibility as a competency even though it is an increasingly desired skillset across computing broadly. Moreover, in the soon-to-be-released CS2023, the Accessibility Knowledge Unit is nested within HCI. This placement fails to highlight the role accessibility plays throughout the CS curriculum, nor does it provide sufficient structure to guide CS educators. Thus, these efforts may not be enough to prepare future tech professionals with adequate accessibility skills to meet growing industry demand. We argue that Accessibility should be its own Knowledge Area within the CS curriculum. We present a set of knowledge units with topics and illustrative learning objectives for an Accessibility Knowledge Area in Computing Education. These objectives were produced through synthesizing computing education and accessible computing literature and discussions among the authors, who are subject matter experts in both computing education and accessibility. In this position paper, we make the case that Accessibility should be incorporated into computer science across the curriculum beyond just HCI, and we demonstrate how the knowledge units spans the different facets of computing. In addition to highlighting the cross-curriculum importance of Accessibility, the presented Knowledge Area outline provides structure for future work in creating teaching resources and guiding curricular integration.