Accurately crude oil price prediction remains challenging so far. Despite the abundant research achievements of crude oil price prediction, most of them emphasize the linear and deterministic modeling, which cannot adequately capture the complex nonlinear characteristics and uncertainties involved, thus impeding further developments in the field. In this study, a novel learning system with the aim of obtaining deterministic and probabilistic predictions is presented to model the nonlinear dynamics in crude oil price, composed by the modules of recurrence analysis, outlier detection, data preprocessing, feature selection, predictive modeling based on deep learning, and system evaluation. In particular, the temporal convolution is developed to perform the feature selection, thus improving the generalization of the system. Additionally, the extensions, including the predictive performance test evaluation, convergence investigation, and sensitivity analysis, are carried out. The experimental simulations show that the proposed system can yield the deterministic and probabilistic predictions with higher accuracy and feasibility compared with the benchmarks considered, further indicating its effectiveness.