Quantile regression is a method of fundamental importance. How to efficiently conduct quantile regression for a large dataset on a distributed system is of great importance. We show that the popularly used one-shot estimation is statistically inefficient if data are not randomly distributed across different workers. To fix the problem, a novel one-step estimation method is developed with the following nice properties. First, the algorithm is communication efficient. That is the communication cost demanded is practically acceptable. Second, the resulting estimator is statistically efficient. That is its asymptotic covariance is the same as that of the global estimator. Third, the estimator is robust against data distribution. That is its consistency is guaranteed even if data are not randomly distributed across different workers. Numerical experiments are provided to corroborate our findings. A real example is also presented for illustration.

International Press of Boston - publishers of scholarly mathematical and scientific journals and books

An interesting application of the link prediction technique is detecting the potential new links in collaboration networks. In this study, we construct collaboration networks based on the co-authorship information of the papers published in 43 statistical journals from 2001 to 2018. We construct training and testing networks according to the timestamps of the papers and construct a classification dataset for link prediction. We calculate 20 similarity indices based on the training network to perform link prediction. Additionally, we consider nodal attributes (institutes and research interests) to develop two novel predictors called the same institute (SIN) and keywords match count (KMC). Several machine-learning classifiers including support vector machine, XGBoost and random forest are implemented to combine all predictors. After incorporating SIN and KMC, we observe that the area under the receiver operating characteristic curve values of all classifiers improved, indicating that SIN and KMC can significantly improve classification accuracy. Finally, we provide collaborative recommendations for researchers based on the proposed model.

Modern statistical analysis often encounters massive datasets with ultrahigh-dimensional features. In this work, we develop a subsampling approach for feature screening with massive datasets. The approach is implemented by repeated subsampling of massive data and can be used for analyzing tasks with memory constraints. To conduct the procedure, we first calculate an R-squared screening measure (and related sample moments) based on subsamples. Second, we consider three methods to combine the local statistics. In addition to the simple average method, we design a jackknife debiased screening measure and an aggregated moment screening measure. Both approaches reduce the bias of the subsampling screening measure and therefore increase the accuracy of the feature screening. Last, we consider a novel sequential sampling method, that is more computationally efficient than the traditional random sampling method. The theoretical properties of the three screening measures under both sampling schemes are rigorously discussed. Finally, we illustrate the usefulness of the proposed method with an airline dataset containing 32.7 million records.