Prioritized experience replay (PER) is an important technique in deep reinforcement learning (DRL). It improves the sampling efficiency of data in various DRL algorithms and achieves great performance. PER uses temporal difference error (TD-error) to measure the value of experiences and adjusts the sampling probability of experiences. Although PER can sample valuable experiences according to the TD-error, freshness is also an important character of experiences. It implicitly reflects the potential value of experiences. Fresh experiences are produced by virtue of the current networks and they are more valuable for updating the current network parameters than the past. The sampling of fresh experiences to train the neural networks can increase the learning speed of the agent, but few algorithms can perform this job efficiently. To solve this issue, a novel experience replay method is proposed in this paper. We first define that the experience freshness is negatively correlated with the number of replays. A new hyper-parameter, the freshness discounted factor $\mu$, is introduced in PER to measure the experience freshness. Further, a novel experience replacement strategy in the replay buffer is proposed to increase the experience replacement efficiency. In our method, the sampling probability of fresh experiences is increased by raising its priority properly. So the algorithm is more likely to choose fresh experiences to train the neural networks during the learning process. We evaluated this method in both discrete control tasks and continuous control tasks via OpenAI Gym. The experimental results show that our method achieves better performance in both modes of operation.

Despite the growing capabilities of the short-term prediction of photovoltaic power, we still face two challenges to longer time-range predictions: error accumulation and long-term time series feature extraction. In order to improve the longer time range prediction accuracy of photovoltaic power, this paper proposes a seq2seq prediction model TCNformer, which outperforms other state-of-the-art (SOTA) algorithms by introducing variable selection (VS), long- and short-term time series feature extraction (LSTFE), and one-step temporal convolutional network (TCN) decoding. A VS module employs correlation analysis and periodic analysis to separate the time series correlation information, LSTFE extracts multiple time series features from time series data, and one-step TCN decoding realizes generative predictions. We demonstrate here that TCNformer has the lowest mean squared error (MSE), mean absolute error (MAE) and mean absolute percentage error (MAPE) in contrast to the other algorithms in the field of the short-term prediction of photovoltaic power, and furthermore, the effectiveness of each module has been verified through ablation experiments.