Sean J. Jones

Sean J. Jones

Auto-generated profile, based on publicly available information

Sean J. Jones

University at Buffalo, Department of Computer Science and Engineering, USA

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 September 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Verified
Journal Article
Open access
JLJiyang Li
Jiyang Li
LHLin Huang
Lin Huang
SSSiddharth Shah
Siddharth Shah
SJSean J. Jones
Sean J. Jones
YJYincheng Jin
Yincheng Jin
ZJZhanpeng Jin
Zhanpeng Jin
Sign language is a natural language widely used by Deaf and hard of hearing (DHH) individuals. Advanced wearables are developed to recognize sign language automatically. However, they are limited by the lack of labeled data, which leads to a small vocabulary and unsatisfactory performance even though laborious efforts are put into data collection. Here we propose SignRing, an IMU-based system that breaks through the traditional data augmentation method, makes use of online videos to generate the virtual IMU (v-IMU) data, and pushes the boundary of wearable-based systems by reaching the vocabulary size of 934 with sentences up to 16 glosses. The v-IMU data is generated by reconstructing 3D hand movements from two-view videos and calculating 3-axis acceleration data, by which we are able to achieve a word error rate (WER) of 6.3% with a mix of half v-IMU and half IMU training data (2339 samples for each), and a WER of 14.7% with 100% v-IMU training data (6048 samples), compared with the baseline performance of the 8.3% WER (trained with 2339 samples of IMU data). We have conducted comparisons between v-IMU and IMU data to demonstrate the reliability and generalizability of the v-IMU data. This interdisciplinary work covers various areas such as wearable sensor development, computer vision techniques, deep learning, and linguistics, which can provide valuable insights to researchers with similar research objectives.