Constrained blackbox optimization is a difficult problem, with most approaches coming from the mathematical programming literature. The statistical literature is sparse, especially in addressing problems with nontrivial constraints. This situation is unfortunate because statistical methods have many attractive properties: global scope, handling noisy objectives, sensitivity analysis, and so forth. To narrow that gap, we propose a combination of response surface modeling, expected improvement, and the augmented Lagrangian numerical optimization framework. This hybrid approach allows the statistical model to think globally and the augmented Lagrangian to act locally. We focus on problems where the constraints are the primary bottleneck, requiring expensive simulation to evaluate and substantial modeling effort to map out. In that context, our hybridization presents a simple yet effective solution that allows existing objective-oriented statistical approaches, like those based on Gaussian process surrogates and expected improvement heuristics, to be applied to the constrained setting with minor modification. This work is motivated by a challenging, real-data benchmark problem from hydrology where, even with a simple linear objective function, learning a nontrivial valid region complicates the search for a global minimum. Supplementary materials for this article are available online.
Bayesian optimization routines may have theoretical convergence results, but determining whether a run has converged in practice can be a subjective task. This paper provides a framework inspired by statistical process control for monitoring an optimization run for convergence. The maximum Expected Improvement (EI) tends to decrease during an optimization run, but decreasing EI is not sufficient for convergence. We consider both a decrease in EI as well as local stability of the variance in order to assess for convergence. The EI process is made more numerically stable through an expected log-normal approximation. An Exponentially Weighted Moving Average control chart is adapted for automated convergence analysis, which allows assessment of stability of both the EI and its variance. The success of the methodology is demonstrated on several examples.
This paper presents an efficient variational inference framework for a family of structured Gaussian process regression network (SGPRN) models. We incorporate auxiliary inducing variables in latent functions and jointly treat both the distributions of the inducing variables and hyper-parameters as variational parameters. Then we take advantage of the collapsed representation of the model and propose structured variational distributions, which enables the decomposability of a tractable variational lower bound and leads to stochastic optimization. Our inference approach is able to model data in which outputs do not share a common input set, and with a computational complexity independent of the size of the inputs and outputs to easily handle datasets with missing values. Finally, we illustrate our approach on both synthetic and real data.
Latent Gaussian processes are widely used in latent variable models because they offer a flexible nonlinear mapping from a low-dimensional latent space to a high-dimensional space via Gaussian processes. However, these models are computationally intensive and do not scale well. In this paper, we propose a temporal categorical model that utilizes latent Gaussian processes, inducing-input approximation, and a regularization framework to model multivariate categorical processes, with and without priors on hyperparameters. We analyze the underlying properties and introduce two variational inference approaches: one based on a Monte Carlo method and the other on the delta method. We also found that latent dynamics tend to collapse into a constant zero, which hinders the reflection of dynamic information. To address this, we propose two strategies to regularize the latent dynamics for better alignment with observation dynamics: introducing a regularization term based on inducing variables and incorporating a scale prior for latent Gaussian processes. Additionally, we propose an efficient and effective stochastic variational inference technique. Finally, we demonstrate our model and inference methods using both synthetic data and real financial data.