31 October 2023 in Proceedings of the ACM on Human-Computer Interaction
Determining the correct orientation of objects can be critical to succeed in tasks like assembly and quality assurance. In particular, near-symmetrical objects may require careful inspection of small visual features to disambiguate their orientation. We propose CADTrack, a digital assistant for providing instructions and support for tasks where the object orientation matters but may be hard to disambiguate with the naked eye. Additionally, we present a deep learning pipeline for tracking the orientation of near-symmetrical objects. In contrast to existing approaches, which require labeled datasets involving laborious data acquisition and annotation processes, CADTrack uses a digital model of the object to generate synthetic data and train a convolutional neural network. Furthermore, we extend the architecture of Mask R-CNN with a confidence prediction branch to avoid errors caused by misleading orientation guidance. We evaluate CADTrack in a user study, comparing our tracking-based instructions to other methods to confirm the benefits of our approach in terms of preference and required effort.
29 June 2026 in Proceedings of the ACM on Human-Computer Interaction
Despite the potential of self-supervised video alignment algorithms for advancing human-computer interaction, they remain largely inaccessible to practitioners without machine learning expertise. To bridge this gap, we introduce VideoAlign, an open-source toolkit designed to facilitate training and integration of video alignment approaches in interactive applications. VideoAlign offers guidance for training models to align videos, detecting and tagging specific events, and performing anomaly detection through an interactive system without requiring machine learning experience. In addition to implementing state-of-the-art alignment techniques, our toolkit introduces a novel Local-Alignment Contrastive (LAC) loss. Unlike global methods that compare entire video sequences, LAC aligns localized segments independently. This capability enables robust matching when video structure or timing varies, which is crucial for real-world interactive applications. We demonstrate how VideoAlign facilitates the creation of a wide variety of interactive applications through three application scenarios: a teacher-support tool for providing efficient video-feedback, a mixed reality application that tracks activity progress in real-time, and an anomaly detection tool to monitor cooking activities.