27 March 2024 in IEEE Transactions on Computational Social Systems
Social networks contributed massive text data generated by users in it, which were crucial in information explosion. These unstructured and ambiguous expressed data may result in difficulty in obtaining available contextual information from it, which can help us gain more accurate insights into user-generated content, user preferences, and topic dynamics within social networks. By training on a large-scale unsupervised corpus and fine-tuning parameters using a limited amount of supervised data, the pretrained language model can effectively capture rich contextual information and achieve excellent performance in numerous downstream tasks of natural language processing (NLP). For the low-resource language such as Tibetan, the distributed representation results of dynamic changes obtained from pretrained language models can effectively alleviate the problem of insufficient labeled data. In order to achieve more effective contextual information and word-level semantic information in Tibetan social media, we collected a large amount of Tibetan language corpus and trained a Tibetan pretrained language model, named as Tibetan-BERT-wwm, by using the whole word masking strategy. Additionally, we apply the model to analyze textual data from social networks to assess its efficacy in capturing user sentiments and news topic in Tibetan social media. In this study, accuracy, precision, recall, and F1 score were used to evaluate its performance. The results showed that the macro-F1 of Tibetan-BERT-wwm in the public dataset TNCC document and title are 75.55% and 64.17%, the self-built sentiment analysis dataset is 70.98%. Compared with other pretrained language models, the Tibetan-BERT-wwm model can capture the semantic information of Tibetan well and improve the Tibetan classification effect.
19 December 2024 in Complex & Intelligent Systems
The advent of ChatGPT and GPT-4 has generated substantial interest in large language model (LLM) research, showcasing remarkable performance in various applications such as conversation systems, machine translation, and research paper summarization. However, their efficacy diminishes when applied to low-resource languages, particularly in academic research contexts like Tibetan. In this study, we trained Tibetan LLaMA (T-LLaMA), a model based on efficient pre-training technology for three downstream tasks: text classification, news text generation and automatic text summarization. To address the lack of corpus, we constructed a Tibetan dataset comprising 2.2 billion characters. Furthermore, we augmented the vocabulary of LLaMA2 from META AI by expanding the Tibetan vocabulary using SentencePiece. Notably, the text classification task attains a state-of-the-art (SOTA) accuracy of 79.8% on a publicly available dataset Tibetan News Classification Corpus. In addition, manual review of 500 generated samples indicates satisfactory results in both news text generation and text summarization tasks. To our knowledge, T-LLaMA stands as the first large-scale language model in Tibetan natural language processing (NLP) with parameters in the billion range. We openly provide our trained models, anticipating that this contribution not only fills gaps in the Tibetan large-scale language model domain but also serves as foundational models for researchers with limited computational resources in the Tibetan NLP community. The T-LLaMA model is available at https://huggingface.co/Pagewood/T-LLaMA.
30 September 2022 in ACM Transactions on Asian and Low-Resource Language Information Processing
Tibetan word segmentation and POS tagging are the primary tasks of Tibetan natural language processing. Most of existing methods of Tibetan word segmentation and POS tagging are based on rules and statistics, which need manual construction of features. In addition, the joint mode has shown stronger capabilities for word segmentation and POS tagging and have received great interests. In this paper, we propose Bi-LSTM+IDCNN+CRF structures, a simple yet effective end-to-end neural network model, for joint Tibetan word segmentation and POS tagging. We conduct step-by-step and joint experiments on the Tibetan datasets. The results demonstrate that the performance of the Bi-LSTM+IDCNN+CRF model is the best regardless of the step-by-step or joint mode. We obtain state-of-the-art performance in the joint tagging mode. The F1 score of the word segmentation task reached 92.31%, and the F1 score of the POS tagging task reached 81.26%.
17 March 2025 in Neurocomputing
1 December 2023 in Tsinghua Science and Technology
Sentence Boundary Disambiguation (SBD) is a preprocessing step for natural language processing. Segmenting text into sentences is essential for Deep Learning (DL) and pretraining language models. Tibetan punctuation marks may involve ambiguity about the sentences' beginnings and endings. Hence, the ambiguous punctuation marks must be distinguished, and the sentence structure must be correctly encoded in language models. This study proposed a component-level Tibetan SBD approach based on the DL model. The models can reduce the error amplification caused by word segmentation and part-of-speech tagging. Although most SBD methods have only considered text on the left side of punctuation marks, this study considers the text on both sides. In this study, 465 669 Tibetan sentences are adopted, and a Bidirectional Long Short-Term Memory (Bi-LSTM) model is used to perform SBD. The experimental results show that the F1-score of the Bi-LSTM model reached 96%, the most efficient among the six models. Experiments are performed on low-resource languages such as Turkish and Romanian, and high-resource languages such as English and German, to verify the models' generalization.
21 November 2024 in ACM Transactions on Asian and Low-Resource Language Information Processing
Tibetan language processing is crucial for preserving its rich cultural heritage and reducing communication barriers between different languages. However, as a low-resource language, the development of Tibetan natural language processing has lagged behind. To address the unique and complex structural information of Tibetan, this article improves the embedding model based on fundamental Tibetan Component-and-Character-and-Word-based Embedding (TCCWE) to enhance the effectiveness of word vector representation. We incorporate position information into the training of Tibetan word vectors, developing models based on components, characters, and their integration. Furthermore, to evaluate the effectiveness of these word vectors, we propose an intrinsic evaluation set, wordsimT, based on k -means clustering. Experimental results demonstrate that the character-based positional vector integration model achieves a Spearman's rank correlation coefficient of 79.99% on the wordsimT benchmark, outperforming the baseline TCCWE model by 1.51%. Additionally, we validate the proposed models in downstream text classification tasks. These findings underscore the importance of incorporating positional information in Tibetan word vectors.