Shang Gao

Shang Gao

Auto-generated profile, based on publicly available information

Shang Gao

Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
3 September 2021 in IEEE Journal of Biomedical and Health Informatics
Verified
Journal Article
Open access
SGShang Gao
Shang Gao
MAMohammed Alawad
Mohammed Alawad
MYM. Todd Young
M. Todd Young
JGJohn Gounley
John Gounley
NSNoah Schaefferkoetter
Noah Schaefferkoetter
GTGeorgia Tourassi
Georgia Tourassi
Bidirectional Encoder Representations from Transformers (BERT) and BERT-based approaches are the current state-of-the-art in many natural language processing (NLP) tasks; however, their application to document classification on long clinical texts is limited. In this work, we introduce four methods to scale BERT, which by default can only handle input sequences up to approximately 400 words long, to perform document classification on clinical texts several thousand words long. We compare these methods against two much simpler architectures - a word-level convolutional neural network and a hierarchical self-attention network - and show that BERT often cannot beat these simpler baselines when classifying MIMIC-III discharge summaries and SEER cancer pathology reports. In our analysis, we show that two key components of BERT - pretraining and WordPiece tokenization - may actually be inhibiting BERT's performance on clinical text classification tasks where the input document is several thousand words long and where correctly identifying labels may depend more on identifying a few key words or phrases rather than understanding the contextual meaning of sequences of text.

1 March 2018 in Journal of the American Medical Informatics Association
Verified
Journal Article
Open access
PDF file
SGShang Gao
Shang Gao
MYMichael T Young
Michael T Young
JQJohn X Qiu
John X Qiu
HYHong-Jun Yoon
Hong-Jun Yoon
JCJames B Christian
James B Christian
ARArvind Ramanathan
Arvind Ramanathan
We explored how a deep learning (DL) approach based on hierarchical attention networks (HANs) can improve model performance for multiple information extraction tasks from unstructured cancer pathology reports compared to conventional methods that do not sufficiently capture syntactic and semantic contexts from free-text documents. Data for our analyses were obtained from 942 deidentified pathology reports collected by the National Cancer Institute Surveillance, Epidemiology, and End Results program. The HAN was implemented for 2 information extraction tasks: (1) primary site, matched to 12 International Classification of Diseases for Oncology topography codes (7 breast, 5 lung primary sites), and (2) histological grade classification, matched to G1–G4. Model performance metrics were compared to conventional machine learning (ML) approaches including naive Bayes, logistic regression, support vector machine, random forest, and extreme gradient boosting, and other DL models, including a recurrent neural network (RNN), a recurrent neural network with attention (RNN w/A), and a convolutional neural network. Our results demonstrate that for both information tasks, HAN performed significantly better compared to the conventional ML and DL techniques. In particular, across the 2 tasks, the mean micro and macroF-scores for the HAN with pretraining were (0.852,0.708), compared to naive Bayes (0.518, 0.213), logistic regression (0.682, 0.453), support vector machine (0.634, 0.434), random forest (0.698, 0.508), extreme gradient boosting (0.696, 0.522), RNN (0.505, 0.301), RNN w/A (0.637, 0.471), and convolutional neural network (0.714, 0.460). HAN-based DL models show promise in information abstraction tasks within unstructured clinical pathology reports.

1 November 2019 in Artificial Intelligence in Medicine
Verified
Journal Article
Open access
SGShang Gao
Shang Gao
JQJohn X. Qiu
John X. Qiu
MAMohammed Alawad
Mohammed Alawad
JHJacob D. Hinkle
Jacob D. Hinkle
NSNoah Schaefferkoetter
Noah Schaefferkoetter
ARArvind Ramanathan
Arvind Ramanathan
We introduce a deep learning architecture, hierarchical self-attention networks (HiSANs), designed for classifying pathology reports and show how its unique architecture leads to a new state-of-the-art in accuracy, faster training, and clear interpretability. We evaluate performance on a corpus of 374,899 pathology reports obtained from the National Cancer Institute's (NCI) Surveillance, Epidemiology, and End Results (SEER) program. Each pathology report is associated with five clinical classification tasks - site, laterality, behavior, histology, and grade. We compare the performance of the HiSAN against other machine learning and deep learning approaches commonly used on medical text data - Naive Bayes, logistic regression, convolutional neural networks, and hierarchical attention networks (the previous state-of-the-art). We show that HiSANs are superior to other machine learning and deep learning text classifiers in both accuracy and macro F-score across all five classification tasks. Compared to the previous state-of-the-art, hierarchical attention networks, HiSANs not only are an order of magnitude faster to train, but also achieve about 1% better relative accuracy and 5% better relative macro F-score.

9 February 2021 in PLOS ONE
Verified
Journal Article
Open access
PDF file
SGShang Gao
Shang Gao
OKOlivera Kotevska
Olivera Kotevska
ASAlexandre Sorokine
Alexandre Sorokine
JCJ. Blair Christian
J. Blair Christian
Named entity recognition (NER) is a key component of many scientific literature mining tasks, such as information retrieval, information extraction, and question answering; however, many modern approaches require large amounts of labeled training data in order to be effective. This severely limits the effectiveness of NER models in applications where expert annotations are difficult and expensive to obtain. In this work, we explore the effectiveness of transfer learning and semi-supervised self-training to improve the performance of NER models in biomedical settings with very limited labeled data (250-2000 labeled samples). We first pre-train a BiLSTM-CRF and a BERT model on a very large general biomedical NER corpus such as MedMentions or Semantic Medline, and then we fine-tune the model on a more specific target NER task that has very limited training data; finally, we apply semi-supervised self-training using unlabeled data to further boost model performance. We show that in NER tasks that focus on common biomedical entity types such as those in the Unified Medical Language System (UMLS), combining transfer learning with self-training enables a NER model such as a BiLSTM-CRF or BERT to obtain similar performance with the same model trained on 3x-8x the amount of labeled data. We further show that our approach can also boost performance in a low-resource application where entities types are more rare and not specifically covered in UMLS.

9 March 2021 in BMC Bioinformatics
Verified
Journal Article
Open access
PDF file
KAKevin De Angeli
Kevin De Angeli
SGShang Gao
Shang Gao
MAMohammed Alawad
Mohammed Alawad
HYHong-Jun Yoon
Hong-Jun Yoon
NSNoah Schaefferkoetter
Noah Schaefferkoetter
GTGeorgia Tourassi
Georgia Tourassi
Background: Automated text classification has many important applications in the clinical setting; however, obtaining labelled data for training machine learning and deep learning models is often difficult and expensive. Active learning techniques may mitigate this challenge by reducing the amount of labelled data required to effectively train a model. In this study, we analyze the effectiveness of 11 active learning algorithms on classifying subsite and histology from cancer pathology reports using a Convolutional Neural Network as the text classification model.Results: We compare the performance of each active learning strategy using two differently sized datasets and two different classification tasks. Our results show that on all tasks and dataset sizes, all active learning strategies except diversity-sampling strategies outperformed random sampling, i.e., no active learning. On our large dataset (15K initial labelled samples, adding 15K additional labelled samples each iteration of active learning), there was no clear winner between the different active learning strategies. On our small dataset (1K initial labelled samples, adding 1K additional labelled samples each iteration of active learning), marginal and ratio uncertainty sampling performed better than all other active learning techniques. We found that compared to random sampling, active learning strongly helps performance on rare classes by focusing on underrepresented classes.Conclusions: Active learning can save annotation cost by helping human annotators efficiently and intelligently select which samples to label. Our results show that a dataset constructed using effective active learning techniques requires less than half the amount of labelled data to achieve the same performance as a dataset constructed using random sampling.

18 January 2022 in IEEE Transactions on Evolutionary Computation
Verified
Journal Article
Closed access
ABAndrew E. Blanchard
Andrew E. Blanchard
MSMayanka Chandra Shekar
Mayanka Chandra Shekar
SGShang Gao
Shang Gao
JGJohn Gounley
John Gounley
ILIsaac Lyngaas
Isaac Lyngaas
DBDebsindhu Bhowmik
Debsindhu Bhowmik
Inspired by the evolution of biological systems, genetic algorithms have been applied to generate solutions for optimization problems in a variety of scientific and engineering disciplines. For a given problem, a suitable genome representation must be defined along with a mutation operator to generate subsequent generations. Unlike natural systems, which display a variety of complex rearrangements (e.g., mobile genetic elements), mutation for genetic algorithms commonly utilizes only random pointwise changes. Furthermore, generalizing beyond pointwise mutations poses a key difficulty as useful genome rearrangements depend on the representation and problem domain. To move beyond the limitations of manually defined pointwise changes, here we propose the use of techniques from masked language models to automatically generate mutations. As a first step, common subsequences within a given population are used to generate a vocabulary. The vocabulary is then used to tokenize each genome. A masked language model is trained on the tokenized data in order to generate possible rearrangements (i.e., mutations). In order to illustrate the proposed strategy, we use string representations of molecules and use a genetic algorithm to optimize for drug-likeness and synthesizability. Our results show that moving beyond random pointwise mutations accelerates genetic algorithm optimization.