The volume and type of streaming data is increasing rapidly, thus, real-time processing scenarios for streaming data have continued to increase. The inherent volatility of streaming data causes large changes in the throughput volatility of stream processing platforms, making evaluations of the maximum sustainable throughput (MST) for such platforms a challenge. To address these problems and improve the low efficiency of manual evaluations, a naïve method is typically used to evaluate the MST of platforms by periodically measuring throughput. The throughput volatility is detected using a skip sliding window that is defined as a bounded sequence in which a limited number of throughput values stored to measure volatility in a skipping fashion in each data growth cycle, which is a fixed time interval with respect to the continuous data rate increases. A skipping sliding window that is used to describe the procedure with which the MST is calculated in an approximate way is based on a sliding window, which is used usually in the context of stateful streaming operators. Then, according to the throughput volatility, the method judges whether the MST has been reached using a latency guarantee. However, because this method has low efficiency and a large error rate, we propose an adaptive MST evaluation method that adds a data-growth factor function to the naïve method cycle that dynamically and adaptively tunes the data rate for each data growth cycle. The experimental results from four open-source benchmarks running on three mainstream stream processing platforms show that the adaptive MST evaluation method has a lower error rate and executes faster compared with the naïve evaluation method. Moreover, the proposed method is insensitive to parameter variations and is suitable for mainstream stream processing platforms.

In the era of big data, as the amount of streaming data continues to increase, stream processing tasks (SPTs) face serious challenges in real-time processing scenarios with low latency and high throughput. However, much of the current literature on the performance of SPTs pays attention to the reactive approach, which cannot well avoid the problem of system crashes due to the inherent performance volatility. In this paper, a novel throughput prediction method based on ExtraTree for SPTs is presented to address these challenges. A volatility detection algorithm was proposed to obtain the reasonable metric values after the performance volatility of SPTs was studied. Moreover, a selection algorithm of regression function was proposed to output the performance values of SPTs under a relative stead state. Furthermore, a ExtraTree-based algorithm was proposed to predict the throughput of SPTs. The experimental results from two open-source benchmarks running on Apache Flink, a popular stream processing system (SPS), indicated that the average of the accuracy and efficiency of the proposed method could achieve 90.535% and 0.835 s/10,000 samples, which proved the effectiveness of the proposed method on the task of predicting the throughput of SPTs.

The use of renewable energy by converting it into heat is an important form of storing energy in a usable form and improving the energy supply flexibility; therefore, the electricity–heating system (EHS) can cope with load fluctuations. However, relevant research is lacking on improving the energy supply limitations by the optimal dispatch of energy flow at the typical EHS, such as the coupled CHP–heat pump–thermal storage system (CCHTS). Based on the study of the energy supply characteristics of the CCHTS for extending the energy supply limitation, this study develops an optimal dispatch method using a heat pump (HP) and the thermal storage (TS) of heating networks to improve the flexibility of the CCHTS and the accommodation capacity of renewable energy. The maximum and minimum energy supply limitation model of the CCHTS and the output power characteristic model are established. Based on the piecewise power supply constraint, the energy flow of the EHS is optimized by using the quadratic programming algorithm. The CCHTS can significantly improve the energy supply flexibility; both coupled combined heat and power (CHP) + HP and coupled CHP + TS can improve the power supply flexibility, but the enhanced effect of CHP + HP is better than that of CHP + TS. An increase of 7.6% in wind power consumption is achieved. The consumption of renewable energy increases by 17.9% in the energy flow optimization results.

Latency prediction for stream processing tasks (SPTs) is a critical issue for stream computing, parameter tuning, load optimization, task scheduling, etc. This study addresses the real-time, volatile, and high-volume nature of streaming workloads to improve latency prediction accuracy. A novel model based on Auto-encoders and Transformers was proposed to address the above challenges. The Auto-encoder is utilized to reduce the dimensionality of the extensive features and generate a compressed feature representation. Subsequently, the Transformer is employed to extract spatio-temporal dependencies and predict the latency of SPTs. Experimental results on six benchmarks (about 260,000 samples in total) demonstrate that compared to state-of-the-art models, the proposed model achieves higher accuracy and lower error rates, with average errors of 0.11% (MAE), 0.02% (RMSE), respectively, and average accuracy of 0.89. Moreover, a time window length of 60 or 80 allows the model to achieve the best average accuracy. Additionally, the ablation study verified that the Transformer significantly improved the prediction performance. These results confirm that the proposed model can accurately predict latency for SPTs, further laying the foundation for intelligent and proactive optimization of stream processing systems (SPSs).