Xiaodong Cui

Xiaodong Cui

Auto-generated profile, based on publicly available information

Xiaodong Cui

IBM T. J. Watson Research Center, Yorktown Heights, NY, USA

Research Focus & Metrics

Academic Career Length

18Years (2009-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 May 2020 in IEEE Signal Processing Magazine
Verified
Journal Article
Closed access
XCXiaodong Cui
Xiaodong Cui
WZWei Zhang
Wei Zhang
UFUlrich Finkler
Ulrich Finkler
GSGeorge Saon
George Saon
MPMichael Picheny
Michael Picheny
DKDavid Kung
David Kung
The past decade has witnessed great progress in automatic speech recognition (ASR) due to advances in deep learning. The improvements in performance can be attributed to both improved models and large-scale training data. The key to training such models is the employment of efficient distributed learning techniques. In this article, we provide an overview of distributed training techniques for deep neural network (DNN) acoustic models used for ASR. Starting with the fundamentals of data parallel stochastic gradient descent (SGD) and ASR acoustic modeling, we investigate various distributed training strategies and their realizations in high-performance computing (HPC) environments with an emphasis on striking a balance between communication and computation. Experiments are carried out on a popular public benchmark to study the convergence, speedup, and recognition performance of the investigated strategies.

Verified
Journal Article
Open access
PDF file
CAConrad Albrecht
Conrad Albrecht
RZRui Zhang
Rui Zhang
XCXiaodong Cui
Xiaodong Cui
MFMarcus Freitag
Marcus Freitag
HHHendrik F. Hamann
Hendrik F. Hamann
SLSiyuan Lu
Siyuan Lu
The growing amount of openly available, meter-scale geospatial vertical aerial imagery and the need of the OpenStreetMap (OSM) project for continuous updates bring the opportunity to use the former to help with the latter, e.g., by leveraging the latest remote sensing data in combination with state-of-the-art computer vision methods to assist the OSM community in labeling work. This article reports our progress to utilize artificial neural networks (ANN) for change detection of OSM data to update the map. Furthermore, we aim at identifying geospatial regions where mappers need to focus on completing the global OSM dataset. Our approach is technically backed by the big geospatial data platform Physical Analytics Integrated Repository and Services (PAIRS). We employ supervised training of deep ANNs from vertical aerial imagery to segment scenes based on OSM map tiles to evaluate the technique quantitatively and qualitatively.

6 September 2009
Verified
Conference Paper
Closed access
XCXiaodong Cui
Xiaodong Cui
JXJian Xue
Jian Xue
BXBing Xiang
Bing Xiang
BZBowen Zhou
Bowen Zhou

16 October 2018 in arXiv
Verified
Preprint
Open access
PDF file
XCXiaodong Cui
Xiaodong Cui
WZWei Zhang
Wei Zhang
ZTZoltán Tüske
Zoltán Tüske
MPMichael Picheny
Michael Picheny
We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve the average fitness of the population. With a back-off strategy in the SGD step and an elitist strategy in the evolution step, it guarantees that the best fitness in the population will never degrade. In addition, individuals in the population optimized with various SGD-based optimizers using distinct hyper-parameters in the SGD step are considered as competing species in a coevolution setting such that the complementarity of the optimizers is also taken into account. The effectiveness of ESGD is demonstrated across multiple applications including speech recognition, image recognition and language modeling, using networks with a variety of deep architectures.

26 October 2021 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Open access
XCXiaodong Cui
Xiaodong Cui
WZWei Zhang
Wei Zhang
AKAbdullah Kayi
Abdullah Kayi
MLMingrui Liu
Mingrui Liu
UFUlrich Finkler
Ulrich Finkler
DKDavid Kung
David Kung
Large-scale distributed training of deep acoustic models plays an important role in today's high-performance automatic speech recognition (ASR). In this paper we investigate a variety of asynchronous decentralized distributed training strategies based on data parallel stochastic gradient descent (SGD) to show their superior performance over the commonly-used synchronous distributed training via allreduce, especially when dealing with large batch sizes. Specifically, we study three variants of asynchronous decentralized parallel SGD (ADPSGD), namely, fixed and randomized communication patterns on a ring as well as a delay-by-one scheme. We introduce a mathematical model of ADPSGD, give its theoretical convergence rate, and compare the empirical convergence behavior and straggler resilience properties of the three variants. Experiments are carried out on an IBM supercomputer for training deep long short-term memory (LSTM) acoustic models on the 2000-hour Switchboard dataset. Recognition and speedup performance of the proposed strategies are evaluated under various training configurations. We show that ADPSGD with fixed and randomized communication patterns cope well with slow learners. When learners are equally fast, ADPSGD with the delay-by-one strategy has the fastest convergence with large batches. In particular, using the delay-by-one strategy, we can train the acoustic model in less than 2 hours using 128 V100 GPUs with competitive word error rates.

20 December 2024 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
XCXiaodong Cui
Xiaodong Cui
ASA F M Saif
A F M Saif
SLSongtao Lu
Songtao Lu
LCLisha Chen
Lisha Chen
TCTianyi Chen
Tianyi Chen
GSGeorge Saon
George Saon
In this paper, we propose a bilevel joint unsupervised and supervised training (BL-JUST) framework for automatic speech recognition. Compared to the conventional pretraining and fine-tuning strategy which is a disconnected twostage process, BL-JUST tries to optimize an acoustic model such that it simultaneously minimizes both the unsupervised and supervised loss functions. Because BL-JUST seeks matched local optima of both loss functions, acoustic representations learned by the acoustic model strike a good balance between being generic and task-specific. We solve the BL-JUST problem using penaltybased bilevel gradient descent and evaluate the trained deep neural network acoustic models on various datasets with a variety of architectures and loss functions. We show that BL-JUST can outperform the widely-used pre-training and fine-tuning strategy and some other popular semi-supervised techniques.