Task scheduling is the key to the full utilization of heterogeneous cloud capabilities for parallel processing of big graphs. Most graph processing systems adopt single-granularity scheduling mechanisms without considering the heterogeneity of the cloud, leading to poor performance.To alleviate it by learning from excellent DAG (Directed Acyclic Graph)-based scheduling techniques accumulated in traditional parallel computing, we first present a streaming DAG-construction heuristic. It transforms a big graph along with graph traversal algorithms to be carried out into a DAG. We then propose a three-phase heterogeneous-aware cluster-scheduling algorithm to schedule the DAG into a heterogeneous cloud for parallel processing. In the first phase, we design a parallel linear clustering algorithm to cluster the DAG into a series of linear clusters with different granularities. In the second phase, we design a heterogeneous-aware load balancing algorithm to map these clusters to different computational nodes of the cloud. In the last phase, we design a task ordering algorithm to assigns these clusters as-early-as-possible start times. Experimental results show that our scheme can generate high-quality schedules and improve the efficiency and performance of parallel processing of big graphs in the heterogeneous cloud.

With the explosive growth of big data in various application areas, it is becoming very important for information management system to know the real-time growth change and the long-term increasing trend of big data. Thus, in this paper, we propose a big data growth scale prediction method based on deep learning intelligence. By analyzing the features of big data in various information systems, we summarize several basic data types which are ubiquitous in all kinds of information systems, and give the calculation method of each data growth scale time series. Considering the complex correlation among various types of data produced in information systems and the demand for data scale prediction, a deep learning prediction model, stacked autoencoders prediction (SAEP), is built. The previous time series prediction only considered one time series, but the SAEP can consider multiple time series simultaneously. We use data growth scale time series with different time interval as training sample sets to study features of big data in information systems, and then we can obtain the parameters of the SAEP model for the expected time interval. In order to solve the training problem of deep neural network, we also give a layer-wise training algorithm whose basic idea is to train all the hidden layers of the SAEP model layer by layer and then fine tune the parameters of the whole network. Repeated experiments show that the prediction performance of the SAEP model is stable and obviously better than the traditional exponential regression prediction method.

Currently, ride-hailing service has tens of millions of registered drivers and hundreds of millions of registered passengers, serving tens of millions of rides per day. Different from social media applications like Facebook and LinkedIn, ride hailing needs to read/write/query a large number of small objects (like photos and audio/video pieces) always fast, so that it can support critical online computations such as face comparison and sentiment analysis on audio/video records. This is of particular importance for ride-hailing service to recognize and avoid potential dangers. Existing object stores (like Haystack and Tectonic) usually store object data in files and place object metadata in a separate key-value store (like RocksDB), which is unsuitable for ride-hailing service mainly because the objects' file-related information is placed together with the object data. This severely affects the I/O performance of object storage: first, for crash consistency, the writes of object data and object metadata must be conducted in separate phases of one transaction, which significantly increases I/O latency; second, the space of deleted objects needs to be reclaimed via compaction, which could sharply lower I/O performance when the system is busy in serving normal read/write requests. This paper describes FrigateBird, a continuously fast object store for ride-hailing service. FrigateBird differs from existing object stores in three aspects. First, we present a metadata/data decoupled service architecture for object storage, where the rich metadata service realizes efficient queries and updates of object metadata, and the raw data service purely performs disk I/O to read/write object data from/to raw disks. Second, we propose a rich metadata structure (called Richmeta) taking the write operation logs as part of object metadata, which allows FrigateBird to simultaneously write the object data to raw disks (without filesystem overhead) and write the metadata to a key-value store, guaranteeing crash consistency by checking whether the transaction is completed and rolling back if not. Third, we design a compaction-free deletion mechanism which can efficiently delete an object by only updating the metadata without involving the data service, so that FrigateBird can efficiently support ride-hailing service's frequent delete operations while avoiding data-migration-caused performance hiccup. Evaluation shows that FrigateBird outperforms the state-of-the-art object stores by up to $3.36\times$ and $27.9\times$ in the mean I/O latency for normal and in-compaction scenarios, respectively.