26 October 2023 in Management Science
Motivated by a long-standing gap between inventory theory and practice, we study online inventory models with unknown cyclic demand distributions. We design provably efficient reinforcement learning (RL) algorithms that leverage the structure of inventory problems to achieve optimal theoretical guarantees that surpass existing results. We apply the standard performance measure in online learning literature, regret, which is defined as the difference between the total expected cost of our policy and the total expected cost of the clairvoyant optimal policy that has full knowledge of the demand distributions a priori. This paper analyzes, in the presence of unknown cyclic demands, both the lost-sales model with zero lead time and the multiproduct backlogging model with positive lead times, fixed joint-ordering costs and order limits. For both models, we first introduce episodic models where inventory is discarded at the end of every cycle, and then build upon these results to analyze the nondiscarding models. Our RL policies HQL and FQL achieve [ ] regret for the episodic lost-sales model and the episodic multiproduct backlogging model, matching the regret lower bound that we prove in this paper. For the nondiscarding models, we construct a bandit learning algorithm on top that governs multiple copies of the previous RL algorithms, named Meta-HQL. Meta-HQL achieves [ ] regret for the nondiscarding lost-sales model with zero lead time, again matching the regret lower bound. For the nondiscarding multiproduct backlogging model, our policy Mimic-QL achieves [ ] regret. Our policies remove the regret dependence on the cardinality of the state-action space for inventory problems, which is an improvement over existing RL algorithms. We conducted experiments with a real sales data set from Rossmann, one of the largest drugstore chains in Europe, and also with a synthetic data set. For both sets of experiments, our policy converges rapidly to the optimal policy and dramatically outperforms the best policy that models demand as independent and identically distributed instead of cyclic. This paper was accepted by J. George Shanthikumar, data science. Funding: X.-Y. Gong was partially supported by an Accenture Fellowship. The work of X.-Y. Gong and D. Simchi-Levi was partially supported by the MIT Data Science Lab. Supplemental Material: The data and online appendices are available at https://doi.org/10.1287/mnsc.2023.4947 .
1 March 2025 in Manufacturing & Service Operations Management
Problem definition: Cloud computing is a multibillion-dollar business that draws substantial capital investments from large companies such as Amazon, Microsoft, and Google. Large cloud providers need to accommodate the growing demand for computing resources while avoiding unnecessary overprovisioning of hardware and operational costs. The underlying decision processes are challenging, as they involve long-term hardware and infrastructure investments under future demand uncertainty. In this paper, we introduce the cloud server deployment problem. One important aspect of the problem is that the infrastructure preparation work has to be planned for before server deployments can take place. Furthermore, a combination of temporal constraints has to be considered together with a variety of physical constraints. Methodology/results: We formulate the underlying optimization problem as a two-stage stochastic program. After carefully examining the demand data and on-the-ground deployment operations, we distill two structural properties on deployment throughput constraints and provide tightness results on a convex relaxation of the second stage. Based on that, we develop efficient cutting-plane methods that exploit the special structure of the problem and can accommodate different risk measures. We test our algorithms with real production traces from Microsoft Azure and demonstrate sizeable cost reductions. We show empirically that the algorithms remain optimal even when the two properties are not fully satisfied. Managerial implications: Cloud supply chain operations were largely executed manually due to their complexity and dynamic nature. In this paper, we show that the key decision processes can be systematically optimized. In particular, we demonstrate that accounting for the stochastic nature of demands results in substantial cost reductions in cloud server deployments. Another benefit of our stochastic optimization approach is the ability to seamlessly integrate configurable risk preferences of cloud providers. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2023.0372 .