David Kung

David Kung

Auto-generated profile, based on publicly available information

David Kung

IBM Research

Research Focus & Metrics

Academic Career Length

22Years (2005-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 October 2021 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Open access
XCXiaodong Cui
Xiaodong Cui
WZWei Zhang
Wei Zhang
AKAbdullah Kayi
Abdullah Kayi
MLMingrui Liu
Mingrui Liu
UFUlrich Finkler
Ulrich Finkler
DKDavid Kung
David Kung
Large-scale distributed training of deep acoustic models plays an important role in today's high-performance automatic speech recognition (ASR). In this paper we investigate a variety of asynchronous decentralized distributed training strategies based on data parallel stochastic gradient descent (SGD) to show their superior performance over the commonly-used synchronous distributed training via allreduce, especially when dealing with large batch sizes. Specifically, we study three variants of asynchronous decentralized parallel SGD (ADPSGD), namely, fixed and randomized communication patterns on a ring as well as a delay-by-one scheme. We introduce a mathematical model of ADPSGD, give its theoretical convergence rate, and compare the empirical convergence behavior and straggler resilience properties of the three variants. Experiments are carried out on an IBM supercomputer for training deep long short-term memory (LSTM) acoustic models on the 2000-hour Switchboard dataset. Recognition and speedup performance of the proposed strategies are evaluated under various training configurations. We show that ADPSGD with fixed and randomized communication patterns cope well with slow learners. When learners are equally fast, ADPSGD with the delay-by-one strategy has the fastest convergence with large batches. In particular, using the delay-by-one strategy, we can train the acoustic model in less than 2 hours using 128 V100 GPUs with competitive word error rates.

17 February 2017 in arXiv
Verified
Preprint
Open access
PDF file
VZVladimir Zolotov
Vladimir Zolotov
DKDavid Kung
David Kung
The paper [1] shows that simple linear classifier can compete with complex deep learning algorithms in text classification applications. Combining bag of words (BoW) and linear classification techniques, fastText [1] attains same or only slightly lower accuracy than deep learning algorithms [2-9] that are orders of magnitude slower. We proved formally that fastText can be transformed into a simpler equivalent classifier, which unlike fastText does not have any hidden layer. We also proved that the necessary and sufficient dimensionality of the word vector embedding space is exactly the number of document classes. These results help constructing more optimal linear text classifiers with guaranteed maximum classification capabilities. The results are proven exactly by pure formal algebraic methods without attracting any empirical data.

10 July 2019 in arXiv
Verified
Preprint
Open access
PDF file
WZWei Zhang
Wei Zhang
XCXiaodong Cui
Xiaodong Cui
UFUlrich Finkler
Ulrich Finkler
GSGeorge Saon
George Saon
AKAbdullah Kayi
Abdullah Kayi
MPMichael Picheny
Michael Picheny
Modern Automatic Speech Recognition (ASR) systems rely on distributed deep learning to for quick training completion. To enable efficient distributed training, it is imperative that the training algorithms can converge with a large mini-batch size. In this work, we discovered that Asynchronous Decentralized Parallel Stochastic Gradient Descent (ADPSGD) can work with much larger batch size than commonly used Synchronous SGD (SSGD) algorithm. On commonly used public SWB-300 and SWB-2000 ASR datasets, ADPSGD can converge with a batch size 3X as large as the one used in SSGD, thus enable training at a much larger scale. Further, we proposed a Hierarchical-ADPSGD (H-ADPSGD) system in which learners on the same computing node construct a super learner via a fast allreduce implementation, and super learners deploy ADPSGD algorithm among themselves. On a 64 Nvidia V100 GPU cluster connected via a 100Gb/s Ethernet network, our system is able to train SWB-2000 to reach a 7.6% WER on the Hub5-2000 Switchboard (SWB) test-set and a 13.2% WER on the Call-home (CH) test-set in 5.2 hours. To the best of our knowledge, this is the fastest ASR training system that attains this level of model accuracy for SWB-2000 task to be ever reported in the literature.

1 September 2005
Verified
Conference Paper
Closed access
RHRenato F. Hentschke
Renato F. Hentschke
JNJagannathan Narasimhan
Jagannathan Narasimhan
DKDavid Kung
David Kung
In this paper, we describe methods to speed up integrated placement and synthesis of high performance designs. We present an analysis of the computation times of various logic synthesis transforms. We then show techniques to reduce computation time based upon judicious selection of gates and nets for the resizing and buffering transforms, respectively. We show that it is possible to obtain savings of up to 28% in CPU time without compromising the quality of the results. For large high performance designs that are quite common these days our savings could translate into several hours of CPU time.

10 April 2019 in arXiv
Verified
Preprint
Open access
PDF file
WZWei Zhang
Wei Zhang
XCXiaodong Cui
Xiaodong Cui
UFUlrich Finkler
Ulrich Finkler
BKBrian Kingsbury
Brian Kingsbury
GSGeorge Saon
George Saon
MPMichael Picheny
Michael Picheny
In this paper, we propose and investigate a variety of distributed deep learning strategies for automatic speech recognition (ASR) and evaluate them with a state-of-the-art Long short-term memory (LSTM) acoustic model on the 2000-hour Switchboard (SWB2000), which is one of the most widely used datasets for ASR performance benchmark. We first investigate what are the proper hyper-parameters (e.g., learning rate) to enable the training with sufficiently large batch size without impairing the model accuracy. We then implement various distributed strategies, including Synchronous (SYNC), Asynchronous Decentralized Parallel SGD (ADPSGD) and the hybrid of the two HYBRID, to study their runtime/accuracy trade-off. We show that we can train the LSTM model using ADPSGD in 14 hours with 16 NVIDIA P100 GPUs to reach a 7.6% WER on the Hub5- 2000 Switchboard (SWB) test set and a 13.1% WER on the CallHome (CH) test set. Furthermore, we can train the model using HYBRID in 11.5 hours with 32 NVIDIA V100 GPUs without loss in accuracy.

4 February 2020 in arXiv
Verified
Preprint
Open access
PDF file
WZWei Zhang
Wei Zhang
XCXiaodong Cui
Xiaodong Cui
AKAbdullah Kayi
Abdullah Kayi
MLMingrui Liu
Mingrui Liu
UFUlrich Finkler
Ulrich Finkler
MPMichael Picheny
Michael Picheny
Decentralized Parallel SGD (D-PSGD) and its asynchronous variant Asynchronous Parallel SGD (AD-PSGD) is a family of distributed learning algorithms that have been demonstrated to perform well for large-scale deep learning tasks. One drawback of (A)D-PSGD is that the spectral gap of the mixing matrix decreases when the number of learners in the system increases, which hampers convergence. In this paper, we investigate techniques to accelerate (A)D-PSGD based training by improving the spectral gap while minimizing the communication cost. We demonstrate the effectiveness of our proposed techniques by running experiments on the 2000-hour Switchboard speech recognition task and the ImageNet computer vision task. On an IBM P9 supercomputer, our system is able to train an LSTM acoustic model in 2.28 hours with 7.5% WER on the Hub5-2000 Switchboard (SWB) test set and 13.3% WER on the CallHome (CH) test set using 64 V100 GPUs and in 1.98 hours with 7.7% WER on SWB and 13.3% WER on CH using 128 V100 GPUs, the fastest training time reported to date.