Sandrine Tornay

Mrs. Sandrine Tornay

orcid

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 May 2019
Verified
Conference Paper
Closed access
STSandrine Tornay
Sandrine Tornay
MRMarzieh Razavi
Marzieh Razavi
NCNecati Cihan Camgoz
Necati Cihan Camgoz
RBRichard Bowden
Richard Bowden
MMMathew Magimai.-Doss
Mathew Magimai.-Doss
Sign language conveys information through multiple channels, such as hand shape, hand movement, and mouthing. Modeling this multichannel information is a highly challenging problem. In this paper, we elucidate the link between spoken language and sign language in terms of production phenomenon and perception phenomenon. Through this link we show that hidden Markov model-based approaches developed to model "articulatory" features for spoken language processing can be exploited to model the multichannel information inherent in sign language for sign language processing.

26 September 2019 in Information
Verified
Journal Article
Open access
PDF file
STSandrine Tornay
Sandrine Tornay
MDMathew Magimai Doss
Mathew Magimai Doss
Communication languages convey information through the use of a set of symbols or units. Typically, this unit is word. When developing language technologies, as words in a language do not have the same prior probability, there may not be sufficient training data for each word to model. Furthermore, the training data may not cover all possible words in the language. Due to these data sparsity and word unit coverage issues, language technologies employ modeling of subword units or subunits, which are based on prior linguistic knowledge. For instance, development of speech technologies such as automatic speech recognition system presume that there exists a phonetic dictionary or at least a writing system for the target language. Such knowledge is not available for all languages in the world. In that direction, this article develops a hidden Markov model-based abstract methodology to extract subword units given only pairwise comparison between utterances (or realizations of words in the mode of communication), i.e., whether two utterances correspond to the same word or not. We validate the proposed methodology through investigations on spoken language and sign language. In the case of spoken language, we demonstrate that the proposed methodology can lead up to discovery of phone set and development of phonetic dictionary. In the case of sign language, we demonstrate how hand movement information can be effectively modeled for sign language processing and synthesized back to gain insight about the derived subunits.

17 January 2025 in IEEE Open Journal of Signal Processing
Verified
Journal Article
Open access
NTNeha Tarigopula
Neha Tarigopula
STSandrine Tornay
Sandrine Tornay
OSOzge Mercanoglu Sincan
Ozge Mercanoglu Sincan
RBRichard Bowden
Richard Bowden
MDMathew Magimai Doss
Mathew Magimai Doss
Sign Language conveys information through multiple channels composed of manual (handshape, hand movement) and non-manual (facial expression, mouthing, body posture) components. Sign language assessment involves giving granular feedback to a learner, in terms of correctness of the manual and non-manual components, aiding the learner's progress. Existing methods rely on handcrafted skeleton-based features for hand movement within a KL-HMM framework to identify errors in manual components. However, modern deep learning models offer powerful spatio-temporal representations for videos to represent hand movement and facial expressions. Despite their success in classification tasks, these representations often struggle to attribute errors to specific sources, such as incorrect handshape, improper movement, or incorrect facial expressions. To address this limitation, we leverage and analyze the spatio-temporal representations from Inflated 3D Convolutional Networks (I3D) and integrate them into the KL-HMM framework to assess sign language videos on both manual and non-manual components. By applying masking and cropping techniques, we isolate and evaluate distinct channels of hand movement, and facial expressions using the I3D model and handshape using the CNN-based model. Our approach outperforms traditional methods based on handcrafted features, as validated through experiments on the SMILE-DSGS dataset, and therefore demonstrates that it can enhance the effectiveness of sign language learning tools.

26 May 2024 in Language Assessment Quarterly
Verified
Journal Article
Open access
FHFranz Holzknecht
Franz Holzknecht
STSandrine Tornay
Sandrine Tornay
ABAlessia Battisti
Alessia Battisti
ABAaron Olaf Batty
Aaron Olaf Batty
KTKatja Tissi
Katja Tissi
SESarah Ebling
Sarah Ebling

30 September 2026 in ACM Transactions on Accessible Computing
Verified
Journal Article
Open access
STSandrine Tornay
Sandrine Tornay
ABAlessia Battisti
Alessia Battisti
ANAlexandre Nanchen
Alexandre Nanchen
FHFranz Holzknecht
Franz Holzknecht
NTNeha Tarigopula
Neha Tarigopula
MDMathew Magimai Doss
Mathew Magimai Doss
In language learning, learners need to develop competence in comprehension and production, both of which are necessary for successful interaction. The use of digital technologies to aid the acquisition of such competencies has proven effective in spoken language learning and is emerging in sign language learning. Most existing sign language learning tools are designed for comprehension acquisition, while the production side involves only self-comparison. However, for learning effective sign language production, good proprioception, spatial reasoning, and observation skills are necessary. There is a need to develop applications that guide learners in various aspects of sign language production. For this reason, we developed an AI-driven, web-based sign language learning application in which sign language production is assessed automatically using neural networks and hidden Markov models. The application guides the learners by providing video feedback and scores at different levels, namely, at (i) the sign level, (ii) the form level (e.g., handshape correctness), and (iii) the spatiotemporal level. This article describes the development process of the application, including its architecture, interface, and AI methods. It also covers the intermediate stages, including evaluation studies of both the UI and the underlying AI model. These studies were conducted with sign language learners and human raters to validate the AI methods underlying the application and feedback designs, and to demonstrate the feasibility of the proposed system. The present study underscores the challenges and key elements to consider when designing such evaluation tools. These elements include the choice of sign language assessment algorithms, the importance of linguistically valid data, the design of effective feedback, the importance of incorporating learners’ perspectives, and utilizing human ratings to validate the tool.