Kenton Lee

Kenton Lee

Auto-generated profile, based on publicly available information

Kenton Lee

Google research

Research Focus & Metrics

Academic Career Length

19Years (2008-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2019
Verified
Conference Paper
Open access
PDF file
JDJacob Devlin
Jacob Devlin
MCMing-Wei Chang
Ming-Wei Chang
KLKenton Lee
Kenton Lee
KTKristina Toutanova
Kristina Toutanova
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

24 May 2019 in arXiv
Verified
Preprint
Open access
PDF file
JDJacob Devlin
Jacob Devlin
MCMing-Wei Chang
Ming-Wei Chang
KLKenton Lee
Kenton Lee
KTKristina Toutanova
Kristina Toutanova
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

1 January 2019
Verified
Conference Paper
Open access
PDF file
CCChristopher Clark
Christopher Clark
KLKenton Lee
Kenton Lee
MCMing-Wei Chang
Ming-Wei Chang
TKTom Kwiatkowski
Tom Kwiatkowski
MCMichael Collins
Michael Collins
KTKristina Toutanova
Kristina Toutanova
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

1 March 2010 in Psychology and Aging
Verified
Journal Article
Closed access
ESElizabeth A. L. Stine-Morrow
Elizabeth A. L. Stine-Morrow
MSMatthew C. Shake
Matthew C. Shake
JMJoseph R. Miles
Joseph R. Miles
KLKenton Lee
Kenton Lee
XGXuefei Gao
Xuefei Gao
GMGeorge McConkie
George McConkie
Previous research has suggested that older readers may self-regulate input during reading differently from the way younger readers do, so as to accommodate age-graded change in processing capacity. For example, older adults may pause more frequently for conceptual integration. Presumably, such an allocation policy would enable older readers to manage the cognitive demands of constructing a semantic representation of the text by off-loading the products of intermediate computations to long-term memory, thus decreasing memory demands as conceptual load increases. This was explicitly tested in 2 experiments measuring word-by-word reading time for sentences in which boundary salience was manipulated but in which semantic content was controlled. With both a computer-based moving-window paradigm that permits only forward eye movements, and an eye-tracking paradigm that allows measurement of regressive eye movements, we found evidence for the proposed tradeoff between early and late wrap-up. Across the 2 experiments, age groups were more similar than different in regulating processing time. However, older adults showed evidence of exaggerated early wrap-up in both experiments. These data are consistent with the notion that readers opportunistically regulate effort and that older readers can use this to good advantage to maintain comprehension.

10 February 2020 in arXiv
Verified
Preprint
Open access
PDF file
KGKelvin Guu
Kelvin Guu
KLKenton Lee
Kenton Lee
ZTZora Tung
Zora Tung
PPPanupong Pasupat
Panupong Pasupat
MCMing-Wei Chang
Ming-Wei Chang
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

23 January 2018 in Journal of Multilingual and Multicultural Development
Verified
Journal Article
Closed access
KLKent Lee
Kent Lee
HLHikyoung Lee
Hikyoung Lee
Universities in East Asia are increasingly adopting English-medium instruction (EMI) to enhance their global competitiveness. Yet little research has been conducted on the impact of English on graduate students in this context, as past research has mostly examined undergraduates. This survey study investigates graduate students at a major university in Korea where EMI is increasingly common, specifically probing their self-perceptions for academic English skills, preferred language learning strategies, and intrinsic motivation to learn and use English. The results indicate correlations between strategy use, efficacy, and intrinsic motivation. They also show limitations in the graduate students’ academic English skills, and shortcomings in their training for English-related tasks. The results show how English and EMI affect graduate students, and also raise some questions about EMI at the graduate school level. Limitations need to be addressed for these students, who will enter academic and professional careers in the global marketplace that demand greater English expertise.