1 January 2017 in Journal of Natural Language Processing
我々は国語研日本語ウェブコーパスと word2vec を用いて単語の分散表現を構築し,その分散表現のデータを nwjc2vec と名付けて公開している.本稿では nwjc2vec を紹介し,nwjc2vec の品質を評価するために行った2種類の評価実験の結果を報告する.第一の評価実験では,単語間類似度の評価として,単語類似度データセットを利用して人間の主観評価とのスピアマン順位相関係数を算出する.第二の評価実験では,タスクに基づく評価として,nwjc2vec を用いて語義曖昧性解消及び回帰型ニューラルネットワークによる言語モデルの構築を行う.どちらの評価実験においても,新聞記事7年分の記事データから構築した分散表現を用いた場合の結果と比較することで,nwjc2vec が高品質であることを示す.
28 February 2023 in ACM Transactions on Asian and Low-Resource Language Information Processing
In recent years, the use of distributed representations has been a fundamental technology for natural language processing. However, Japanese has multiple compound words, and often we must compare the meanings of a word and a compound word. Moreover, word boundaries in Japanese are unspecific because Japanese does not have delimiters between words, e.g., “ぶどう狩り” (grape picking) is one word according to one dictionary, whereas “ぶどう” and “狩り” are different words according to another dictionary. This study describes an attempt to compose word embeddings of a compound word from its constituent words in Japanese. We used “short unit” and “long unit,” both of which are the units of terms in UniDic—a Japanese dictionary compiled by the National Institute for Japanese Language and Linguistics—for constituent and compound words, respectively. Furthermore, we composed a word embedding of a compound word from the word embeddings of two constituent words using a neural network. The training data for the word embedding of compound words was created using a corpus generated by concatenating the corpora divided by constituent and compound words. We propose using linguistic knowledge for compositing word embedding to demonstrate how it improves the composition performance. We compared cosine similarity between composed and correct word embeddings of compound words to assess models with and without linguistic knowledge. Furthermore, we evaluated our methods by the ranking of synonyms using a thesaurus. We compared several frameworks and algorithms that use three types of linguistic knowledge—semantic patterns, parts of speech patterns, and compositionality score—and then investigated which linguistic knowledge improves the composition performance. The experiments demonstrated that the multitask models with the classification task of the parts of speech patterns and the estimation task of compositionality scores achieved high performances.