As an increasing number of traditional applications migrated to the cloud, achieving resource management and performance optimization in such a dynamic and uncertain environment becomes a big challenge for cloud-based application providers. In particular, job scheduling is a non-trivial task, which is responsible for allocating massive job requests submitted by users to the most suitable resources and satisfying user QoS requirements as much as possible. Inspired by recent success of using deep reinforcement learning techniques to solve AI control problems, in this paper, we propose an intelligent QoS-aware job scheduling framework for application providers. A deep reinforcement learning-based job scheduler is the key component of the framework. It is able to learn to make appropriate online job-to-VM decisions for continuous job requests directly from its experiences without any prior knowledge. Experimental results using synthetic workloads and real-world NASA workload traces show that compared with other baseline solutions, our proposed job scheduling approach can efficiently reduce average job response time (e.g., reduced by 40.4% compared with the best baseline for NASA traces), guarantee the QoS at a high level (e.g., job success rate is higher than 93% for all simulated changing workload scenarios), and adapt to different workload conditions.
Cloud computing is an emerging paradigm which provides a flexible and diversified trading market for Infrastructure‐as‐a‐Service (IaaS) providers, Software‐as‐a‐Service (SaaS) providers, and cloud‐based application customers. Taking the perspective of SaaS providers, they offer various SaaS services using rental cloud resources supplied by IaaS providers to their end users. In order to maximize their utility, the best behavioural strategy is to reduce renting expenses as much as possible while providing sufficient processing capacity to meet customer demands. In reality, public IaaS providers such as Amazon offer different types of virtual machine (VM) instances with different pricing models. Moreover, service requests from customers always change as time goes by. In such heterogeneous and changing environments, how to realize application auto‐scaling becomes increasingly significant for SaaS providers. In this paper, we first formulate this problem and then propose a Q‐learning based self‐adaptive renting plan generation approach to help SaaS providers make efficient IaaS facilities adjustment decisions dynamically. Through a series of experiments and simulation, we evaluate the auto‐scaling approach under different market conditions and compare it with two other resource allocation strategies. Experimental results show that our approach could automatically generate optimal renting policies for the SaaS provider in the long run.
Despite having played a significant role in the Industry 4.0 era, the Internet of Things is currently faced with the challenge of how to ingest large-scale heterogeneous and multi-type device data. In response to this problem we present a heterogeneous device data ingestion model for an industrial big data platform. The model includes device templates and four strategies for data synchronization, data slicing, data splitting and data indexing, respectively. We can ingest device data from multiple sources with this heterogeneous device data ingestion model, which has been verified on our industrial big data platform. In addition, we present a case study on device data-based scenario analysis of industrial big data.
With the development and maturity of cloud storage, it has attracted a large number of users. Although cloud users do not need to concern themselves with the infrastructure used for storage, thus saving on equipment and maintenance costs, the sheer volume of data still generates significant cloud storage usage costs, which motivates cloud storage users to look for ways to further save costs. In this article, we analyze the whole process of using cloud storage to exhaustively explore opportunities, motivations, and challenges of cost optimization from user perspectives. Then we provide a comprehensive taxonomy and summary of recent advances in terms of storage efficiency (i.e., cost optimization by improving storage efficiency), cloud storage services (i.e., cost optimization by leveraging the features of cloud storage services), and emerging storage paradigms (i.e., cost optimization by leveraging emerging storage paradigms like edge storage). Finally, we present future directions for cost optimization from user perspectives and present our conclusion. This article offers a thorough survey of recent advances focusing on how to optimize the cost of using cloud storage for cloud users, and it has an opportunity to attract a broad audience in the cost-effective cloud storage market.