Joan Jo Jo Sim

Joan Jo Jo Sim

Auto-generated profile, based on publicly available information

Joan Jo Jo Sim

Faculty of Social Sciences and Humanities, Universiti Malaysia Sarawak, Sarawak, Malaysia

Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 January 2026 in Data in Brief
Verified
Journal Article
Open access
YCYuan Ting Chong
Yuan Ting Chong
YRYessane Shrrie Nagendhra Rao
Yessane Shrrie Nagendhra Rao
RKREHMAN ULLAH KHAN
REHMAN ULLAH KHAN
CTChee Siong Teh
Chee Siong Teh
MBMohamad Hardyman Barawi
Mohamad Hardyman Barawi
JSJoan Jo Jo Sim
Joan Jo Jo Sim
Sign languages all around the world are unique and diverse. Each sign language shows the differences in cultural nuances of its origin locale giving it is distinctive nature. Thus, despite the positive outcomes of sign language recognition and translation research that has been widely conducted worldwide, there are still notable limitations to each system which are mainly caused by data limitations. The sign language recognition and translation research in Malaysia especially has been set back by the limited size and nature of datasets available that are concurrent with current technological developments. The current datasets available for Malaysian Sign Language (BIM - Bahasa Isyarat Malaysia) are small and limited to fingerspelling of alphanumeric characters and several dynamic words and short phrases. However, given the continuous nature of the sign language communication, these data are not enough to properly train machine learning models to recognize and translate continuous real-world signs. Therefore, in order to address this issue, we introduce a dynamic BIM dataset which comprises of video, gloss, and translation data consisting of alphanumeric characters, dynamic words and short phrases, and continuous sentences. The dataset is split into two versions. The first version, BIM-SSD-V1 dataset comprises of 4,858 parallel video (RGB frames), gloss, and translation data while the second version, BIM-SSD-V2 dataset comprises of 3,143 parallel video (RGB frames), keypoints and gloss data for recognition purposes, and 4,900 parallel gloss and translation data for translation purposes. The raw videos are also available in the dataset. The dataset was developed and compiled with the help of the Deaf and Hard-of-Hearing community. This process also included the development of a Sign Language Module (translations for the video and gloss data) to assist in the development of the dataset. The image and video data were collected using smartphones and the respective gloss annotations for the data were prepared with the help of a BIM expert. The data collection process was designed to reflect everyday communication scenarios by incorporating varied sentence constructions, repeated signing instances, and recordings under different backgrounds and contextual conditions to introduce data-level variability relevant to real-world use. The total number of participants involved in the data collection process was four. There are also four samples for every character, word, phrase or sentence in the Sign Language Module. The dataset can mainly be reused by researchers who would like to conduct sign language recognition and translation research using the Sign-to-Gloss-to-Text framework. However, the dataset is not limited to only one framework and can be used for other sign language recognition and translation research frameworks accordingly.

19 June 2026 in SSRN Electronic Journal
Verified
Preprint
Open access
YCYuan Ting Chong
Yuan Ting Chong
RKREHMAN ULLAH KHAN
REHMAN ULLAH KHAN
RNRao Yessane Shrrie Nagendhra
Rao Yessane Shrrie Nagendhra
CTChee Siong Teh
Chee Siong Teh
MBMohamad Hardyman Barawi
Mohamad Hardyman Barawi
JSJoan Jo Jo Sim
Joan Jo Jo Sim
Continuous Sign Language Recognition (CSLR) is essential for natural sign language communication but remains challenging due to strong signer-dependence, reliance on RGB appearance features, and limited generalisation validation. This study proposes a new training method, termed Two-Stage Signer Adaptation Training (TSAT) to improve signer generalisation and support real-time deployment for Malaysian Sign Language (BIM, Bahasa Isyarat Malaysia). TSAT was designed within a Sign-to-Gloss (S2G) framework and evaluated using stickman-based visual representations to reduce sensitivity to visual appearance variations. Experiments were conducted on the BIM-SSD-V2 dataset with 3143 samples and an additional Unseen-Signers dataset from new individuals. Results show that RGB-based model achieved competitive performance on seen signers but failed to generalise unseen signers, highlighting severe signer dependence through original RGB frames. When applied stickman representations with TSAT, this enables incremental adaptation across signer distributions and improve unseen signer performance from 88.50% to 41% Word Error Rate (WER). The proposed methods offer a practical and reproduceable training strategy for robust CSLR systems, particularly for low-resource sign languages and real-time applications. Introduce TSAT to improve unseen signers’ generalisation.Reduce signer-dependence by decoupling training from RGB appearance features through stickman-based visual representations.Demonstrate improved unseen signers’ recognition on low-resource sign languages, BIM.