Named entity recognition (NER) is one fundamental task in natural language processing, which is typically addressed by neural condition random field (CRF) models, regarding the task as a sequence labeling problem. Sentence-level information has been shown positive for the task. Equipped with sophisticated neural structures such as long-short term memory network (LSTM), implicit sentence-level global information can be exploited fully, and has also been demonstrated effective in previous studies. In this work, we propose a new method for better learning of these sentence-level features in an explicit manner. Concretely, we suggest an auxiliary task, namely sentence-level named type prediction (i.e., determining whether a sentence includes a certain kind of named type), to supervise the feature representation learning globally. We conduct experiments on six benchmark datasets of various languages to evaluate our method. The results show that our final model is highly effective, resulting in significant improvements and leading to highly competitive results on all datasets.

The lack of human annotations has been one of the main obstacles for neural named entity recognition in low-resource domains. To address this problem, there have been many efforts on automatically generating silver annotations according to domain-specific dictionaries. However, the information of domain dictionaries is usually limited, and the generated annotations may be noisy which poses significant challenges on learning effective models. In this work, we try to alleviate these issues by introducing a dictionary-guided graph attention model. First, domain-specific dictionaries are utilized to extract entity mention candidates by a graph matching algorithm, which can capture word patterns of domain entities. Furthermore, a word-mention interactive graph is leveraged to integrate the semantic and boundary information of entities into their context. We evaluated our model on the biomedical-domain datasets of recognizing chemical and disease entities, namely BC5CDR and NCBI disease corpora. The results show that our model outperforms several state-of-the-art models with different methodologies, such as feature-based models (e.g., BANNER), ensemble models (e.g., CollaboNet), multi-task learning models (e.g., MTM-CW), dictionary-based models (e.g., AutoNER). Moreover, the performance of our model is also comparable with BioBERT that owns huge parameters and needs large-scale pre-training.

Emojis, utilizing visual means, mimic human facial expressions and postures to convey emotions and opinions. They are widely used in social media platforms such as Sina Weibo, and have become a crucial feature for sentiment analysis. However, existing approaches often treat emojis as special symbols or convert them into text labels, thereby neglecting the rich visual information of emojis. We propose a novel multimodal information integration model for emoji microblog sentiment analysis. To effectively leverage the emoji visual information, the model employs a text-emoji visual mutual attention mechanism. Experiments on a manually annotated microblog dataset show that compared to the baseline models without incorporating emoji visual information, the proposed model achieves improvements of 1.37% in macro F1 score and 2.30% in accuracy, respectively. To facilitate the related research, our corpus will be publicly available at https://github.com/yx100/Emojis/blob/main/weibo-emojis-annotation.

Normalizing disease names is a crucial task for biomedical and healthcare domains. Previous work explored various approaches, including rules, machine learning and deep learning, which focused on only one approach or one model. In this study, we systematically investigated the performances of various neural models and the effects of different features. Our investigation was performed on two benchmark datasets, namely the NCBI disease corpus and the BioCreative V Chemical Disease Relation (BC5CDR) corpus. The convolutional neural network (CNN) performed the best (F1 90.11%) in the NCBI disease corpus and the attention neural network (Attention) performed the best (F1 90.78%) in the BC5CDR corpus. Compared with the state-of-the-art system, DNorm, our models improved the F1s by 1.74% and 0.86% respectively. In terms of features, character information could improve the F1 by about 0.5-1.0% while sentence information worsened the F1 by about 3-4%. Moreover, we proposed a novel approach for pre-training models, which improved the F1 by up to 9%. The CNN and Attention models are comparable in the task of disease name normalization while the recurrent neural network performs much worse. In addition, character information and pre-training techniques are helpful for this task while sentence information hurts the performance. Our proposed models and pre-training approach can be easily adapted to the normalization task for any other type of entities. Our source code is available at: https://github.com/yx100/EntityNorm.

Recent works have increasingly adopted pre-trained language models, such as BERT, to model technical semantics for patent relevance assessment. However, existing truncation and divide-merge strategies, used to handle input length constraints, results in feature loss and semantic isolation. This limits the effectiveness of pre-trained language models in capturing long sequence semantics for patent relevance assessment. Because patent relevance is evaluated based on local sequence semantics, this paper proposes a patent-oriented bidirectional encoder representation from transformers (P-BERT) to enable the semantic modeling ability of long sequences. The characteristics of P-BERT lie in a prefix sequence compression method, a soft position embedding method and a data augmentation method. The first one can retrain more technical features within the input constraints via transforming full sequences into partial sequences while the second one reconstructs local sequence semantics for partial sequences. As for the last one, it contains sample and feature augmentation to improve the generalization and robustness. Due to the scarcity of training corpus for patent relevance assessment, a dual constraint strategy is proposed to generate a supervised corpus, DualCons. Experimental results demonstrate that DualCons can help BERT obtain the best ranking performance compared with other strategies. Model evaluations show the effectiveness of P-BERT. It reaches an improvement of 2%-6% in recall-oriented metrics and an improvement of 1%-5% in accuracy-oriented indicators over the state-of-the-art baselines. All those would promote the engineering applications of pre-trained language models in patent relevance assessment.