With the most advanced natural language processing and artificial intelligence approaches, effective summarization of long and multi-topic documents—such as academic papers—for readers from different domains still remains a challenge. To address this, we introduce ConceptEVA, a mixed-initiative approach to generate, evaluate, and customize summaries for long and multi-topic documents. ConceptEVA incorporates a custom multi-task longformer encoder decoder to summarize longer documents. Interactive visualizations of document concepts as a network reflecting both semantic relatedness and co-occurrence help users focus on concepts of interest. The user can select these concepts and automatically update the summary to emphasize them. We present two iterations of ConceptEVA evaluated through an expert review and a within-subjects study. We find that participants’ satisfaction with customized summaries through ConceptEVA is higher than their own manually-generated summary, while incorporating critique into the summaries proved challenging. Based on our findings, we make recommendations for designing summarization systems incorporating mixed-initiative interactions.
Analysis of large, high-dimensional, and heterogeneous datasets is challenging as no one technique is suitable for visualizing and clustering such data in order to make sense of the underlying information. For instance, heterogeneous logs detailing machine repair and maintenance in an organization often need to be analyzed to diagnose errors and identify abnormal patterns, formalize root-cause analyses, and plan preventive maintenance. Such real-world datasets are also beset by issues such as inconsistent and/or missing entries. To conduct an effective diagnosis, it is important to extract and understand patterns from the data with support from analytic algorithms (e.g., finding that certain kinds of machine complaints occur more in the summer) while involving the human-in-the-loop. To address these challenges, we adopt existing techniques for dimensionality reduction (DR) and clustering of numerical, categorical, and text data dimensions, and introduce a visual analytics approach that uses multiple coordinated views to connect DR + clustering results across each kind of the data dimension stated. To help analysts label the clusters, each clustering view is supplemented with techniques and visualizations that contrast a cluster of interest with the rest of the dataset. Our approach assists analysts to make sense of machine maintenance logs and their errors. Then the gained insights help them carry out preventive maintenance. We illustrate and evaluate our approach through use cases and expert studies respectively, and discuss generalization of the approach to other heterogeneous data.
Exercise benefits people not only in physical health but also in mental health. Compared to individual workouts, many people prefer group exercise because of the additional peer support like guidance from experienced peers, and socializing benefits like the opportunities to meet new friends when exercising with others as a group. However, during the pandemic, many group exercise sessions were shifted from in-person to online through video conferencing platforms. Such change significantly decreased the exercise feedback and social signals that participants could receive during the exercise, such as the immediate self-checking with a studio mirror, the group movement calibration with co-exercisers, and the sense of social presence (i.e., the existence of other co-exercisers) in the small community. To address these challenges, we present ZoombaTogether, a video platform add-on that enhances the remote Zumba dancing experience with on-the-fly pose detection and visualization techniques. It tracks and evaluates how well remote dancers follow the instructor with minor latency and provide them with visual feedback to simulate a joint exercise experience with group members. ZoombaTogether visualizes the participants’ performance at both individual and group levels to help them understand their movement similarity to the instructor and other dancers with a set of extensible visualization modules. Meanwhile, ZoombaTogether addresses the participants’ privacy concerns by supporting on-device computer vision-based body posture estimation so that they could keep their personal video private from other group members. In this work, we provide a solution to use visual feedback as a communication channel for exercise participants and demonstrate the possibility of enhancing the online group exercise experience. We encourage the community to carry on our work and explore solutions to the challenges of missing social signals in online group exercises.
As machine learning (ML) gains wider adoption in real-world applications, the validation of ML models becomes fundamental for its productization, particularly in safety-critical applications. Recently, data slice finding has emerged as a popular method for validating ML models, but it requires additional metadata or cross-modal embeddings for the slices to be interpretable. We propose ConceptSlicer, an integrated workflow that facilitates the slicing of computer vision models using visual concepts. This approach breaks down the image dataset into interpretable visual concepts, serving as metadata in the slice finding process. Our system offers insights into model issues and enables a deeper understanding of computer vision models’ strengths and weaknesses. We evaluate ConceptSlicer through interviews with eight domain experts and machine learning practitioners, and fine-tune the ML models based on their feedback. Our study also highlights varied attitudes towards large foundational models, encouraging contemplation of the challenges and opportunities presented by this technological advancement.
Preserving embodied cultural heritage like martial arts requires more than simple digitization: a single technique may be named in a title, demonstrated in a video, and reworked in an exhibition, yet metadata alone cannot connect these scattered records with the overarching concepts and context they share. To address this gap, this work refines the Martial Arts Ontology (MAon) into an archive-oriented content model that resolves technical-unit granularity and terminology variation across heterogeneous records. We introduce DAExt, a role-based descriptive layer that characterizes records by their archival roles, enabling the same record to carry different meanings across archival contexts while exposing selected roles as Linked Art-compatible patterns. We assess our solution in two ways: first, by modeling multimodal records in a digital archive as ontology instances across four archival roles, and second, by using a preliminary coverage test of 6,968 entries from the Chinese Martial Arts Dictionary as a scope check for the refined ontology. The resulting cases show that our approach can describe heterogeneous records across archival contexts while preserving local martial-arts semantics as well as supporting interoperable archival description. Meanwhile, the coverage test indicates broad lexical reach and identifies remaining gaps for future work.