Obtaining high-quality embeddings of out-of-vocabularies (OOVs) and low-frequency words is a challenge in natural language processing (NLP). To efficiently estimate the embeddings of OOVs and low-frequency words, we propose a new method that uses the dictionary to estimate the embeddings of OOVs and low-frequency words. More specifically, the explanatory note of an entry in dictionaries accurately describes the semantics of the corresponding word. Naturally, we adopt the sentence representation model to extract the semantics of the explanatory note and regard the semantics as the embedding of the corresponding word. We design a new sentence representation model to encode sentences to extract the semantics from the explanatory notes of entries more efficiently. Based on the assumption that the higher quality of word embeddings will lead to better performance, we design an extrinsic experiment to evaluate the quality of low-frequency words’ embeddings. The experimental results show that the embeddings of low-frequency words estimated by our proposed method have higher quality. In addition, both intrinsic and extrinsic experiments show that our proposed sentence representation model can represent the semantics of sentences well.
29 October 2021 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Sentence representation is an important research hotspot in natural language processing (NLP) since it can map the semantics of sentences into semantics vectors, thereby effectively solving complex semantics computing problems. Recently, sentence representations are mainly obtained by indirect means. Specifically, for sentence representations obtained by unsupervised means, they are often calculated by the weighted sum of embeddings of tokens in sentences; for sentence representations obtained by self-supervised or supervised means, they are often derived from intermediate encodings of sentences in prediction tasks. For example, Google's BERT and MUSE respectively use the embedding of [CLS] in the next sentence prediction task and intermediate encodings of sentences in the translation bridge task as sentence representations. In this paper, we use the observed semantics increment feature of sentences to directly model the semantics function of sentences. To be able to use the existing neural network language model to approximate the semantics function, we first implement the first-order Taylor expansion on the semantics function to obtain a difference semantics model and then add it to BERT as a subtask to perform self-supervised fine-tuning. Finally, we get a new sentence representation model S-BERT. S-BERT achieves the state-of-the-art performance on many datasets in Chinese, English, and Vietnamese.