Students' interest in game development often leads departments to attempt to put together a stand-alone game development course. We have used both Unity and Unreal for just such a course. Unity and Unreal are discussed here in the context of how viable they are and how they compare in terms of usefulness for teaching game development. We also present them in context of high level overviews of other game engines and discuss which factors add to a more effective game engine for teaching game development. This paper is intended to help answer some of the questions asked by those without experience in the field who find themselves developing stand-alone game development courses.

We present research on how the perception of intelligent systems can be influenced by early experiences of machine performance, and how explainability potentially helps users develop an accurate understanding of system capabilities. Using a custom video analysis system with AI-assisted activity recognition, we studied whether presenting explanatory information for system outputs affects user perception of the system. In this experiment, some participants encountered AI weaknesses early, while others encountered the same limitations later in the study. The difference in ordering had a significant impact on user understanding of the system and the ability to detect AI strengths and weaknesses, and the addition of explanations was not enough to counteract the strong effects of early impressions. The results demonstrate the importance of first impressions with intelligent systems and motivate the need for improved methods of intervention to combat automation bias.

We consider the following activity recognition task: given a video, infer the set of activities being performed in the video and assign each frame to an activity. This task can be solved using modern deep learning architectures based on neural networks or conventional classifiers such as linear models and decision trees. While neural networks exhibit superior predictive performance as compared with decision trees and linear models, they are also uninterpretable and less explainable. We address thisaccuracy‐explanability gapusing a novel framework that feeds the output of a deep neural network to an interpretable, tractable probabilistic model called dynamic cutset networks, and performs joint reasoning over the two to answer questions. The neural network helps achieve high accuracy while dynamic cutset networks because of their polytime probabilistic reasoning capabilities make the system more explainable. We demonstrate the efficacy of our approach by using it to build three prototype systems that solve human‐machine tasks having varying levels of difficulty using cooking videos as an accessible domain. We describe high‐level technical details and key lessons learned in our human subjects evaluations of these systems.

Conducting data analysis tasks rarely occur in isolation. Especially in intelligence analysis scenarios where different experts contribute knowledge to a shared understanding, members must communicate how insights develop to establish common ground among collaborators. The use of provenance to communicate analytic sensemaking carries promise by describing the interactions and summarizing the steps taken to reach insights. Yet, no universal guidelines exist for communicating provenance in different settings. Our work focuses on the presentation of provenance information and the resulting conclusions reached and strategies used by new analysts. In an open-ended, 30-minute, textual exploration scenario, we qualitatively compare how adding different types of provenance information (specifically data coverage and interaction history) affects analysts' confidence in conclusions developed, propensity to repeat work, filtering of data, identification of relevant information, and typical investigation strategies. We see that data coverage (i.e., what was interacted with) provides provenance information without limiting individual investigation freedom. On the other hand, while interaction history (i.e., when something was interacted with) does not significantly encourage more mimicry, it does take more time to comfortably understand, as represented by less confident conclusions and less relevant information-gathering behaviors. Our results contribute empirical data towards understanding how provenance summarizations can influence analysis behaviors.

Jeremy Block, Yu-Peng Chen, Abhilash Budharapu, Lisa Anthony, Bonnie Dorr. Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems. 2023.

Research in interpretable machine learning proposes different computational and human subject approaches to evaluate model saliency explanations. These approaches measure different qualities of explanations to achieve diverse goals in designing interpretable machine learning systems. In this paper, we propose a human attention benchmark for image and text domains using multi-layer human attention masks aggregated from multiple human annotators. We then present an evaluation study to evaluate model saliency explanations obtained using Grad-cam and LIME techniques. We demonstrate our benchmark's utility for quantitative evaluation of model explanations by comparing it with human subjective ratings and ground-truth single-layer segmentation masks evaluations. Our study results show that our threshold agnostic evaluation method with the human attention baseline is more effective than single-layer object segmentation masks to ground truth. Our experiments also reveal user biases in the subjective rating of model saliency explanations.