In this paper, we study the black box optimization problem under the Polyak–Lojasiewicz (PL) condition, assuming that the objective function is not just smooth, but has higher smoothness. By using “kernel-based” approximations instead of the exact gradient in the Stochastic Gradient Descent method, we improve the best-known results of convergence in the class of gradient-free algorithms solving problems under the PL condition. We generalize our results to the case where a zeroth-order oracle returns a function value at a point with some adversarial noise. We verify our theoretical results on the example of solving a system of nonlinear equations.
28 October 2024 in Journal of Optimization Theory and Applications
A possible method for estimating the unknown parameters of dynamic models described by differential-algebraic equations is considered. The parameters are estimated using the observations of a mathematical model. The parameter values are found by minimizing a criterion written as the sum of the squared deviations of the values of the state vector’s coordinates from their exact counterparts obtained through measurements at different time instants. Parallelepiped-type constraints are imposed on the parameter values. For solving the optimization problem, a mini-batch method of adaptive random search is proposed, which further develops the ideas of optimization methods used in machine learning. This method is applied for solving three model problems, and the results are compared with those obtained by gradient optimization methods of machine learning procedures and also with those obtained by metaheuristic algorithms.
This paper studies non-smooth problems of convex stochastic optimization. Using the smoothing technique based on the replacement of the function value at the considered point by the averaged function value over a ball (in l1-norm or l2-norm) of a small radius centered at this point, and then the original problem is reduced to a smooth problem (whose Lipschitz constant of the gradient is inversely proportional to the radius of the ball). An essential property of the smoothing used is the possibility of calculating an unbiased estimation of the gradient of a smoothed function based only on realizations of the original function. The obtained smooth stochastic optimization problem is proposed to be solved in a distributed federated learning architecture (the problem is solved in parallel: nodes make local steps, e.g. stochastic gradient descent, then communicate—all with all, then all this is repeated). The goal of the article is to build on the basis of modern achievements in the field of gradient–free non-smooth optimization and in the field of federated learning gradient-free methods for solving problems of non-smooth stochastic optimization in the architecture of federated learning.