17 May 2023 in Proceedings of the ACM on Human-Computer Interaction
Enabling gaze interaction in real-time on handheld mobile devices has attracted significant attention in recent years. An increasing number of research projects have focused on sophisticated appearance-based deep learning models to enhance the precision of gaze estimation on smartphones. This inspires important research questions, including how the gaze can be used in a real-time application, and what type of gaze interaction methods are preferable under dynamic conditions in terms of both user acceptance and delivering reliable performance. To address these questions, we design four types of gaze scrolling techniques: three explicit technique based on Gaze Gesture, Dwell time, and Pursuit; and one implicit technique based on reading speed to support touch-free, page-scrolling on a reading application. We conduct a 20-participant user study under both sitting and walking settings and our results reveal that Gaze Gesture and Dwell time-based interfaces are more robust while walking and Gaze Gesture has achieved consistently good scores on usability while not causing high cognitive workload.
5 May 2022 in Methods in Ecology and Evolution
When recorders are used to survey acoustically conspicuous species, identification calls of the target species in recordings is essential for estimating density and abundance. We investigate how well deep neural networks identify vocalisations consisting ofphrasesof varying lengths, each containing a variable number ofsyllables. We use recordings of Hainan gibbonNomascus hainanusvocalisations to develop and test the methods. We propose two methods for exploiting the two‐level structure of such data. The first combines convolutional neural network (CNN) models with a hidden Markov model (HMM) and the second uses a convolutional recurrent neural network (CRNN). Both models learn acoustic features of syllables via a CNN and temporal correlations of syllables into phrases either via an HMM or recurrent network. We compare their performance to commonly used CNNs LeNet and VGGNet, and support vector machine (SVM). We also propose a dynamic programming method to evaluate how well phrases are predicted. This is useful for evaluating performance when vocalisations are labelled by phrases, not syllables. Our methods perform substantially better than the commonly used methods when applied to the gibbon acoustic recordings. The CRNN has anF‐score of 90% on phrase prediction, which is 18% higher than the best of the SVM or LeNet and VGGNet methods. HMM post‐processing raised theF‐score of these last three methods to as much as 87%. The number of phrases is overestimated by CNNs and SVM, leading to error rates between 49% and 54%. With HMM, these error rates can be reduced to 0.4% at the lowest. Similarly, the error rate of CRNN's prediction is no more than 0.5%. CRNNs are better at identifying phrases of varying lengths composed of a varying number of syllables than simpler CNN or SVM models. We find a CRNN model to be best at this task, with a CNN combined with an HMM performing almost as well. We recommend that these kinds of models are used for species whose vocalisations are structured into phrases of varying lengths.
1 July 2024 in Biometrics
Passive acoustic monitoring can be an effective way of monitoring wildlife populations that are acoustically active but difficult to survey visually, but identifying target species calls in recordings is non-trivial. Machine learning (ML) techniques can do detection quickly but may miss calls and produce false positives, i.e., misidentify calls from other sources as being from the target species. While abundance estimation methods can address the former issue effectively, methods to deal with false positives are under-investigated. We propose an acoustic spatial capture-recapture (ASCR) method that deals with false positives by treating species identity as a latent variable. Individual-level outputs from ML techniques are treated as random variables whose distributions depend on the latent identity. This gives rise to a mixture model likelihood that we maximize to estimate call density. We compare our method to existing methods by applying it to an ASCR survey of frogs and simulated acoustic surveys of gibbons based on real gibbon acoustic data. Estimates from our method are closer to ASCR applied to the dataset without false positives than those from a widely used false positive “correction factor” method. Simulations show our method to have bias close to zero and accurate coverage probabilities and to perform substantially better than ASCR without accounting for false positives.
16 July 2026 in Journal of Agricultural, Biological and Environmental Statistics