Xiaohan Wei

Xiaohan Wei

Auto-generated profile, based on publicly available information

Xiaohan Wei

University of Southern California

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
31 May 2018 in IEEE Transactions on Information Theory
Verified
Journal Article
Closed access
LGLarry Goldstein
Larry Goldstein
SMStanislav Minsker
Stanislav Minsker
XWXiaohan Wei
Xiaohan Wei
We study high-dimensional signal recovery from non-linear measurements with design vectors having elliptically symmetric distribution. Special attention is devoted to the situation when the unknown signal belongs to a set of low statistical complexity, while both the measurements and the design vectors are heavy-tailed. We propose and analyze a new estimator that adapts to the structure of the problem, while being robust both to the possible model misspecification characterized by arbitrary non-linearity of the measurements as well as to data corruption modeled by the heavy-tailed distributions. Moreover, this estimator has low computational complexity. Our results are expressed in the form of exponential concentration inequalities for the error of the proposed estimator. On the technical side, our proofs rely on the generic chaining methods, and illustrate the power of this approach for statistical applications. Theory is supported by numerical experiments demonstrating that our estimator outperforms existing alternatives when data is heavy-tailed.

14 August 2021
Verified
Conference Paper
Open access
VGVipul Gupta
Vipul Gupta
DCDhruv Choudhary
Dhruv Choudhary
PTPeter Tang
Peter Tang
XWXiaohan Wei
Xiaohan Wei
Xing WangXing Wang
Xing Wang
MMMichael W. Mahoney
Michael W. Mahoney
In this paper, we consider hybrid parallelism---a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP)---to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during the forward and backward propagation, respectively. This is done by identifying and updating only the most relevant neurons of the neural network for each training sample in the data. We evaluate DCT on publicly available natural language processing and recommender models and datasets, as well as recommendation systems used in production at Facebook. DCT reduces communication by at least 100x and 20x during DP and MP, respectively. The algorithm has been deployed in production, and it improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.

12 May 2017 in IEEE/ACM Transactions on Networking
Verified
Journal Article
Closed access
XWXiaohan Wei
Xiaohan Wei
MNMichael J. Neely
Michael J. Neely
This paper considers a cost minimization problem for data centers with $N$ servers and randomly arriving service requests. A central router decides which server to use for each new request. Each server has three types of states (active, idle, and setup) with different costs and time durations. The servers operate asynchronously over their own states and can choose one of multiple sleep modes when idle. We develop an online distributed control algorithm so that each server makes its own decisions. The request queues are bounded and the overall time average cost is near optimal with probability 1. First the algorithm does not need probability information for the arrival rate or job sizes. Finally, an improved algorithm that uses a single queue is developed via a ``virtualization'' technique, which is shown to provide the same (near optimal) costs. Simulation experiments on a real data center traffic trace demonstrate the efficiency of our algorithm compared with other existing algorithms.

19 May 2021 in IEEE Micro
Verified
Journal Article
Closed access
ZDZhaoxia Summer Deng
Zhaoxia Summer Deng
JPJong-Soo Park
Jong-Soo Park
PTPing Tak Peter Tang
Ping Tak Peter Tang
HLHaixin Liu
Haixin Liu
JYJie Yang
Jie Yang
MSMikhail Smelyanskiy
Mikhail Smelyanskiy
Tremendous success of machine learning (ML) and the unabated growth in model complexity motivated many ML-specific designs in hardware architectures to speed up the model inference. While these architectures are diverse, highly optimized low-precision arithmetic is a component shared by most. Nevertheless, recommender systems important to Facebook's personalization services are demanding and complex: They must serve billions of users per month responsively with low latency while maintaining high prediction accuracy. Do these low-precision architectures work well with our production recommendation systems They do. But not without significant effort. In this paper, we share our search strategies to adapt reference recommendation models to low-precision hardware, our optimization of low-precision compute kernels, and the tool chain to maintain our models' accuracy throughout their lifespan. We believe our lessons from the trenches can promote better co-design between hardware architecture and software engineering and advance the state of the art of ML in industry.

21 May 2018 in Information and Inference: A Journal of the IMA
Verified
Journal Article
Closed access
PDF file
LGLarry Goldstein
Larry Goldstein
XWXiaohan Wei
Xiaohan Wei
Performance guarantees for estimates of unknowns in nonlinear compressed sensing models under non-Gaussian measurements can be achieved through the use of distributional characteristics that are sensitive to the distance to normality, and which in particular return the value of zero under Gaussian or linear sensing. The use of these characteristics, or discrepancies, improves some previous results in this area by relaxing conditions and tightening performance bounds. In addition, these characteristics are tractable to compute when Gaussian sensing is corrupted by either additive errors or mixing.

14 August 2021
Verified
Conference Paper
Open access
YHYuzhen Huang
Yuzhen Huang
XWXiaohan Wei
Xiaohan Wei
XWXing Wang
Xing Wang
JYJiyan Yang
Jiyan Yang
BSBor-Yiing Su
Bor-Yiing Su
JLJack Langman
Jack Langman
Neural network based recommendation models are widely used to power many internet-scale applications including product recommendation and feed ranking. As the models become more complex and more training data is required during training, improving the training scalability of these recommendation models becomes an urgent need. However, improving the scalability without sacrificing the model quality is challenging. In this paper, we conduct an in-depth analysis of the scalability bottleneck in existing training architecture on large scale CPU clusters. Based on these observations, we propose a new training architecture called Hierarchical Training, which exploits both data parallelism and model parallelism for the neural network part of the model within a group. We implement hierarchical training with a two-layer design: a tagging system that decides the operator placement and a net transformation system that materializes the training plans, and integrate hierarchical training into existing training stack. We propose several optimizations to improve the scalability of hierarchical training including model architecture optimization, communication compression, and various system-level improvements. Extensive experiments at massive scale demonstrate that hierarchical training can speed up distributed recommendation model training by 1.9x without model quality drop.