Hand gestures provide an alternate interaction modality for blind users and can be supported using commodity smartwatches without requiring specialized sensors. The enabling technology is an accurate gesture recognition algorithm, but almost all algorithms are designed for sighted users. Our study shows that blind user gestures are considerably different from sighted users, rendering current recognition algorithms unsuitable. Blind user gestures have high inter-user variance, making learning gesture patterns difficult without large-scale training data. Instead, we design a gesture recognition algorithm that works on a 3D representation of the gesture trajectory, capturing motion in free space. Our insight is to extract a micro-movement in the gesture that is user-invariant and use this micro-movement for gesture classification. To this end, we develop an ensemble classifier that combines image classification with geometric properties of the gesture. Our evaluation demonstrates a 92% classification accuracy, surpassing the next best state-of-the-art which has an accuracy of 82%.
In this paper, we present AccessWear, a system that improves the accessibility of smartphone touchscreen interactions for blind users using smartwatch gestures. Our system design is human-centered, namely, it incorporates the design goals that were learned from a formative user study with 9 blind participants. The formative study showed that blind users liked the idea of using smartwatch gestures as an alternative: 4 participants liked that when using smart-watch gestures, they did not have to bring their expensive phones out in public and 6 participants liked that smart-watch gestures can be performed with one-hand, as the other hand is usually occupied in holding a cane or a guide dog. Even though there are several advantages to smartwatch gestures, our study also shows that gestures performed by blind users have different patterns compared to sighted users, making gesture recognition more challenging. To this end, AccessWear makes two contributions. The first is a gesture recognition system that works specifically for blind users that is lightweight and does not require per-person training. The second is a near-zero-effort gesture replacement system that does not require any changes to the original application. AccessWear uses input virtualization techniques so that a given gesture can replace the touchscreen input seamlessly. We implement AccessWear on an Android smartphone and Android watch. We perform a quantitative and qualitative study with 8 blind participants. Our study shows that AccessWear can recognize gestures with a 92% accuracy and the end-to-end latency when using an alternate gesture was 53 msec on average. The qualitative study shows that when participants perform a task, consisting of a series of gestures, the system is robust, does not have perceived delays, and does not add physical or mental load on the users.
31 January 2023 in ACM Transactions on Computing for Healthcare
According to the World Health Organisation (WHO), 235 million people suffer from respiratory illnesses which causes four million deaths annually. Regular lung health monitoring can lead to prognoses about deteriorating lung health conditions. This article presents our system SpiroMask that retrofits a microphone in consumer-grade masks (N95 and cloth masks) for continuous lung health monitoring. We evaluate our approach on 48 participants (including 14 with lung health issues) and find that we can estimate parameters such as lung volume and respiration rate within the approved error range by the American Thoracic Society (ATS). Further, we show that our approach is robust to sensor placement inside the mask.