Data center (DC) plays an important role to support services, such as e-commerce and cloud computing. The resulting energy consumption from this growing market has drawn significant attention, and noticeably almost half of the energy cost is used to cool the DC to a particular temperature. It is thus an critical operational challenge to curb the cooling energy cost without sacrificing the thermal safety of a DC. The existing solutions typically follow a two-step approach, in which the system is first modeled based on expert knowledge and, thus, the operational actions are determined with heuristics and/or best practices. These approaches are often hard to generalize and might result in suboptimal performances due to intrinsic model errors for large-scale systems. In this paper, we propose optimizing the DC cooling control via the emerging deep reinforcement learning (DRL) framework. Compared to the existing approaches, our solution lends itself an end-to-end cooling control algorithm (CCA) via an off-policy offline version of the deep deterministic policy gradient (DDPG) algorithm, in which an evaluation network is trained to predict the DC energy cost along with resulting cooling effects, and a policy network is trained to gauge optimized control settings. Moreover, we introduce a de-underestimation (DUE) validation mechanism for the critic network to reduce the potential underestimation of the risk caused by neural approximation. Our proposed algorithm is evaluated on an EnergyPlus simulation platform and on a real data trace collected from the National Super Computing Centre (NSCC) of Singapore. The resulting numerical results show that the proposed CCA can achieve up to 11% cooling cost reduction on the simulation platform compared with a manually configured baseline control algorithm. In the trace-based study of conservative nature, the proposed algorithm can achieve about 15% cooling energy savings on the NSCC data trace. Our pioneering approach can shed new light on the application of DRL to optimize and automate DC operations and management, potentially revolutionizing digital infrastructure management with intelligence.

The converter steelmaking process smelts hot metal to liquid steel and occupies an important position in industry. The composition of liquid steel at the endpoint is an essential quality index, including the concentrations of multiple elements, such as carbon, silicon, and manganese. Accurately predicting endpoint composition is the basis of production optimization. Hence, a multichannel diffusion graph convolutional network (MCDGCN) is presented in this article. Unlike conventional models, the developed MCDGCN describes the converter steelmaking process as a graph to exploit the correlations among element concentrations for an accurate endpoint composition prediction. We also develop a unique K-hop diffusion method to extract the globally consistent information over the graph for predicting each element. The proposed method addresses the composition prediction task for a realistic converter steelmaking process. To the best of our knowledge, this is the first time that up to 15 elements of liquid steel are covered and predicted to present a comprehensive process model. Compared with six benchmark models, MCDGCN presents state-of-the-art results, i.e., an average R ² of 0.8475 and an average MAE of 0.0189, which shows that the correlation mining of graph deep learning can indeed improve the prediction performance for endpoint composition.

Model-based reinforcement learning (MBRL) has been proposed as a promising alternative solution to tackle the high sampling cost challenge in the canonical RL, by leveraging a system dynamics model to generate synthetic data for policy training purpose. The MBRL framework, nevertheless, is inherently limited by the convoluted process of jointly optimizing control policy, learning system dynamics, and sampling data from two sources controlled by complicated hyperparameters. As such, the training process involves overwhelmingly manual tuning and is prohibitively costly. In this research, we propose a ``reinforcement on reinforcement'' (RoR) architecture to decompose the convoluted tasks into two decoupled layers of RL. The inner layer is the canonical MBRL training process which is formulated as a Markov decision process, called training process environment (TPE). The outer layer serves as an RL agent, called intelligent trainer, to learn an optimal hyperparameter configuration for the inner TPE. This decomposition approach provides much-needed flexibility to implement different trainer designs, referred to ``train the trainer.'' In our research, we propose and optimize two alternative trainer designs: 1) an unihead trainer and 2) a multihead trainer. Our proposed RoR framework is evaluated for five tasks in the OpenAI gym. Compared with three other baseline methods, our proposed intelligent trainer methods have a competitive performance in autotuning capability, with up to 56% expected sampling cost saving without knowing the best parameter configurations in advance. The proposed trainer framework can be easily extended to tasks that require costly hyperparameter tuning.

Web services serve as the cornerstone for Internet to render webpages. The initial rendering latency of webpages, depending on a subset of critical objects in the webpage, is one of the key metrics for web services. In this work, we propose to identify this set of critical objects in a systematic manner and cache them with a higher priority to reduce the initial rendering time. We first conduct a measurement study on a mainstream content delivery network provider to motivate our study, suggesting that, in status quo, not all cached objects are critical and only a small portion of critical objects is cached. It follows that we model the critical-object aware caching scheme as a constrained optimization problem. Using the stochastic optimization framework, we decompose the problem into a set of one-shot optimization problems, proved to be NP-hard. We then develop two greedy algorithms, with different computation complexity and the same performance bound. Finally, we integrate the resulted approximation algorithms into an online algorithm. Through trace-based simulations, we verify that our proposed algorithm can reduce the service latency and network traffic with a higher cache hit ratio.