Ningxia wolfberry is the only wolfberry product with medicinal value in China. However, the nutritional elements, active ingredients, and economic value of the wolfberry vary considerably among different origins in Ningxia. It is difficult to determine the origin of wolfberry by traditional methods due to the same variety, similar origins, and external characteristics. In the study, we have for the first time used a multi-task residual fully convolutional network (MRes-FCN) under Bayesian optimized architecture for imaging from visible-near-infrared (Vis-NIR, 400–1000 nm) and near-infrared (NIR-1700 nm) hyperspectral imaging (HSI) technology to establish a classification model for near geographic origin of Ningxia wolfberries (Zhongning, Guyuan, Tongxin, and Huinong). The denoising auto-encoder (DAE) was used to generate augmented data, then principal component analysis (PCA) was combined with gray level co-occurrence matrix (GLCM) to extract the texture features. Finally, three datasets (HSI, DAE, and texture) were added to the multi-task model. The reshaped data were up-sampled using transposed convolution. After data-sparse processing, the backbone network was imported to train the model. The results showed that the MRes-FCN model exhibited excellent performance, with the accuracies of the full spectrum and optimum characteristic spectrum of 95.54% and 96.43%, respectively. This study has demonstrated that the MRes-FCN model based on Bayesian optimization and DAE data augmentation strategy may be used to identify the near geographical origin of wolfberries.