Deep learning-based language models (LMs) have transcended the gold standard (human baseline) of SQuAD 1.1 and GLUE benchmarks in April and July 2019, respectively. As of 2022, the top five LMs on the SuperGLUE benchmark leaderboard have exceeded the gold standard. Even people with good general knowledge will struggle to solve problems in specialized fields such as medicine and artificial intelligence. Just as humans learn specialized knowledge through bachelor’s, master’s, and doctoral courses, LMs also require a process to develop the ability to understand domain specific knowledge. Thus, this study proposes SciDeBERTa and SciDeBERTa(CS) as a pre-trained LM (PLM) specialized in the science technology domain. We further pretrain the DeBERTa, which was trained with a general corpus, with the science technology domain corpus. Experiments verified that SciDeBERTa(CS) continually pre-trained in the computer science domain achieved 3.53% and 2.17% higher accuracies than SciBERT and S2ORC-SciBERT, respectively, which are science technology domain specialized PLMs, in the task of recognizing entity names in SciERC dataset. In the JRE task of the SciERC dataset, SciDeBERTa(CS) demonstrated a 6.7% higher performance than baseline SCIIE. In the Genia dataset, SciDeBERTa achieved the best performance compared to S2ORC-SciBERT, SciBERT, BERT, DeBERTa and SciDeBERTa(CS). Furthermore, re-initialization technology and optimizers after Adam were explored during fine-tuning to verify the language understanding of PLMs.
The emergence of large language models (LLMs) has marked a significant milestone in the evolution of natural language processing. With the expanded use of LLMs in multiple fields, the development of domain-specific pre-trained language models (PLMs) has become a natural progression and requirement. Developing domain-specific PLMs requires careful design, considering not only differences in training methods but also various factors such as the type of training data and hyperparameters. This paper proposes MediBioDeBERTa, a specialized language model (LM) for biomedical applications. First, we present several practical analyses and methods for improving the performance of LMs in specialized domains. As the initial step, we developed SciDeBERTa v2, an LM specialized in the scientific domain. In the SciERC dataset evaluation, SciDeBERTa v2 achieves the state-of-the-art model performance in the named entity recognition (NER) task. We then provide an in-depth analysis of the datasets and training methods used in the biomedical field. Based on these analyses, MediBioDeBERTa, was continually trained on SciDeBERTa v2 to specialize in the biomedical domain. Utilizing the biomedical language understanding and reasoning benchmark (BLURB), we analyzed factors that degrade task performance and proposed additional improvement methods based on intermediate fine-tuning. The results demonstrate improved performance in three categories: named entity recognition (NER), semantic similarity (SS), and question-answering (QnA), as well as in the ChemProt relation extraction (RE) task on BLURB, compared with existing state-of-the-art LMs.
5 August 2020 in IEEE Transactions on Visualization and Computer Graphics
This article presents a real-time bokeh rendering technique that splats pre-computed sprites but takes dynamic visibilities and intrinsic appearances into account at runtime. To attain alias-free looks without excessive sampling on a lens, the visibilities of strong highlights are densely sampled using rasterization, while regular objects are sparsely sampled using conventional defocus-blur rendering. The intrinsic appearance is dynamically transformed from a precomputed look-up table, which encodes radial aberrations against image distances in a compact 2D texture. Our solution can render complex bokeh effects without undersampling artifacts in real time, and greatly improve the photorealism of defocus-blur rendering.
This study suggests a method to select core data that will be helpful for machine learning. Specifically, we form a two-dimensional distribution based on the similarity of the training data and compose grids with fixed ratios on the distribution. In each grid, we select data based on the distribution consistency (DC) of the target class data and examine how it affects the classifier. We use CIFAR-10 for the experiment and set various grid ratios from 0.5 to 0.005. The influences of these variables were analyzed with the use of different training data sizes selected based on high-DC, low-DC (inverse of high DC), and random (no criteria) selections. As a result, the average point accuracy at 0.95% (±0.65) and the point accuracy at 1.54% (±0.59) improved for the grid configurations of 0.008 and 0.005, respectively. These outcomes justify an improved performance compared with that of the existing approach (data distribution search). In this study, we confirmed that the learning performance improved when the training data were selected for very small grid and high-DC settings.
5 September 2022 in Multimedia Tools and Applications
본 연구는 자동적이고 지능적인 AI 개발에 요구되는 human-in-the-loop (HITL) 과정의 단축을 위한 것으로, 학습에 긍정적 영향을 미치는 데이터 선정 방법을 다룬다. 이를 위해 학습 데이터의 유사성을 중심으로 2차원 분포를 형성하고, 일정한 비율로 격자를 형성한다. 각 격자 내에서 동일한 클래스 데이터의 분포 일관성을 기준으로 데이터를 선정하기 위하여 LSTRf 기법을 응용하여 적용한다. 이를 통해 선정된 데이터는 CNN 기반의 분류기를 통해 학습하고 그 성능을 평가한다. CIFAR-10을 활용하여 실험하였으며, 격자의 크기, 1회 연산에서의 선택되는 데이터 수를 다양화하여 학습에 미치는 효과를 평가하였다. 선정된 학습 데이터는 임의로 선정한 동일한 크기의 데이터와 비교하였다. 그 결과 격자의 크기가 작을수록(0.008과 0.005 크기) 학습에 긍정적으로 영향을 미치는 것을 확인하였고, 1회 연산에서 선정된 개수가 많을수록 전체적으로 우수해짐을 확인할 수 있었다.