18 April 2022 in International Journal of Human–Computer Interaction
Touchless interfaces allow surgeons to control medical imaging systems autonomously while maintaining total asepsis in the Operating Room. This is specially relevant as it applies to the recent outbreak of COVID-19 disease. The choice of best gestures/commands for such interfaces is a critical step that determines the overall efficiency of surgeon-computer interaction. In this regard, usability metrics such as task completion time, memorability and error rate have a long-standing as potential entities in determining the best gestures. In addition, previous works concerned with this problem utilized qualitative measures to identify the best gestures. In this work, we hypothesize that there is a correlation between gestures’ qualitative properties and their usability metrics. In this regard, we conducted a user experiment with language experts to quantify gestures’ properties (v). Next, we developed a gesture-based system that facilitates surgeons to control the medical imaging software in a touchless manner. Next, a usability study was conducted with neurosurgeons and the standard usability metrics (u) were measured in a systematic manner. Lastly, multi-variate correlation analysis was used to find the relations between u and v. Statistical analysis showed that the v scores were significantly correlated with the usability metrics with an and p < 0.05. Once the correlation is established, we can utilize either gestures’ qualitative properties or usability metrics to identify the best set of gestures.
1 June 2020 in IEEE Transactions on Human-Machine Systems
Choosing adequate gestures for touchless interfaces is a challenging task that has a direct impact on human-computer interaction. Such gestures are commonly determined by the designer, ad-hoc, rule-based, or agreement-based methods. Previous approaches to assess agreement grouped the gestures into equivalence classes and ignored the integral properties that are shared between them. In this article, we propose a generalized framework that inherently incorporates the gesture descriptors into the agreement analysis. In contrast to previous approaches, we represent gestures using binary description vectors and allow them to be partially similar. In this context, we introduce a new metric referred to as soft agreement rate (SAR) to measure the level of agreement and provide a mathematical justification for this metric. Furthermore, we perform computational experiments to study the behavior of SAR and demonstrate that existing agreement metrics are a special case of our approach. Our method is evaluated and tested through a guessability study conducted with a group of neurosurgeons. Nevertheless, our formulation can be applied to any other user-elicitation study. Results show that the level of agreement obtained by SAR is 2.64 times higher than the previous metrics. Finally, we show that our approach complements the existing agreement techniques by generating an artificial lexicon based on the most agreed properties.
8 March 2021 in IEEE Robotics and Automation Letters
Planning object manipulation policies based on tactile observations alone is a challenging task due to the multi-factorial variances in the measured point cloud (e.g. sparsity, missing regions, rotation, etc.) and the limited sensory information available through tactile sensing. Nevertheless, mainstream grasp planners are designed for well-structured point cloud data, and lack the crucial ability to plan grasps in unexplored regions that are common during tactile sampling. For these reasons, grasp detection using tactile observations remains a challenging problem. To address this limitation, we propose a novel framework that utilizes a support set of CAD models to augment the tactile observations in order to facilitate object recognition, visualization, and developing manipulation policies solely using tactile samples. To cope with the noise and sparsity of tactile observations, we propose uGPIS, a surface reconstruction method that utilizes the occupancy possibility function and the Gaussian Process Regression to recover the underlying surface from tactile point clouds. Then, we complete the partially observed tactile point cloud using the prior knowledge obtained from the support set of full CAD models that allows grasp detection to be performed using enriched geometric details. Our experimental results on a physical simulation show that our method can successfully combine the prior knowledge from the database to enhance the grasp success rate.
23 September 2022 in IEEE Transactions on Artificial Intelligence
Zero-shot learning (ZSL) is a paradigm in transfer learning that aims to recognize unknown categories by having a mere description of them. The problem of ZSL has been thoroughly studied in the domain of static object recognition; however, ZSL for dynamic events (zero-shot event recognition, ZSER) such as activities and gestures has hardly been investigated. In this context, this article addresses ZSER by relying on semantic attributes of events to transfer the learned knowledge from seen classes to unseen ones. First, we utilized the Amazon Mechanical Turk platform to create the first attribute-based gesture dataset, referred to as zero shot gestural learning (ZSGL), comprising the categories present in MSRC and Italian gesture datasets. Overall, our ZSGL dataset consisted of 26 categories, 65 discriminative attributes, and 16 attribute annotations and 400 examples per category. We used trainable recurrent networks and 3-D convolutional neural networks (CNNs) to learn the spatiotemporal features. Next, we propose a simple yet effective end-to-end approach for ZSER, referred to as joint sequential semantic encoder (JSSE), to explore temporal patterns, to efficiently represent events in the latent space, and to simultaneously optimize for both the semantic and classification tasks. We evaluate our model on ZSGL and two action datasets (UCF and HMDB), and compared the performance of JSSE against several existing baselines under four experimental conditions: 1) within-category , 2) across-category , 3) closed-set , and 4) open-set . Results show that JSSE considerably outperforms ( $p< 0.05$ ) other approaches and performs favorably for both the datasets under all experimental conditions.