We introduce an efficient iterative algorithm, intended for various least squares problems, based on a design of experiments perspective. The algorithm, called orthogonalizing EM (OEM), works for ordinary least squares (OLS) and can be easily extended to penalized least squares. The main idea of the procedure is to orthogonalize a design matrix by adding new rows and then solve the original problem by embedding the augmented design in a missing data framework. We establish several attractive theoretical properties concerning OEM. For the OLS with a singular regression matrix, an OEM sequence converges to the Moore-Penrose generalized inverse-based least squares estimator. For ordinary and penalized least squares with various penalties, it converges to a point having grouping coherence for fully aliased regression matrices. Convergence and the convergence rate of the algorithm are examined. Finally, we demonstrate that OEM is highly efficient for large-scale least squares and penalized least squares problems, and is considerably faster than competing methods when n is much larger than p. Supplementary materials for this article are available online.
This article introduces an interpolation-based method, called the reconstruction approach, for nonparametric regression. Based on the fact that interpolation usually has negligible errors compared to statistical estimation, the reconstruction approach uses an interpolator to parameterize the regression function with its values at finite knots, and then estimates these values by (regularized) least squares. Some popular methods including kernel ridge regression can be viewed as its special cases. It is shown that, the reconstruction idea not only provides different angles to look into existing methods, but also produces new effective experimental design and estimation methods for nonparametric models. In particular, for some methods of complexity , where n is the sample size, this approach provides effective surrogates with much less computational burden. This point makes it very suitable for large datasets. Supplementary materials for this article are available online.
Multi-level and sequential computer experiments are commonly used to study complex systems in engineering and science. Suitable designs for such experiments require nested structures. Because a nested Latin hypercube design (NLHD) (Qian (2009)) can be constructed only when the run size in each larger design is a multiple of that in a smaller one, the use of NLHDs faces some limitations in practice. In this paper, we propose a random sampling procedure for constructing flexible NLHDs, which have no such a restriction. We also develop an efficient sequential algorithm to search for the optimal NLHDs with respect to some space-filling criteria. Simulation results show that our methods can yield flexible NLHDs with good space-filling properties.
23 December 2021 in Applied Stochastic Models in Business and Industry