Soil organic matter (SOM) is a key factor for evaluating soil fertility. Rapidly monitoring organic matter content in desert soil can provide a scientific basis for the rational development and utilization of reserve arable land resources. Although spectral inversion accuracy for SOM under laboratory-controlled conditions is high, it is time-consuming and costly compared to the in situ spectroscopic determination method. However, in situ spectroscopy causes losses in accuracy due to interference from external environmental factors (e.g., the surface roughness of soil, changes in weather conditions, atmospheric water vapor, etc.). Therefore, reducing or removing the interference of external environmental factors to improve the accuracy of in situ spectroscopy for estimating SOM is challenging. In this study, visible and near-infrared (Vis-NIR) in situ spectral data were collected from 135 topsoil (0–20 cm) samples in a desert area of northwestern China, and organic matter content was measured. Three spectral pre-processing methods—the standard normal transform (SNV), reciprocal logarithm (log(1/R)) and normalization (NOR)—combined with three feature variable selection methods—the particle swarm algorithm (PSO), ant colony algorithm (ACO) and simulated annealing (SA) algorithm—were used to filter the spectral feature bands of SOM, and then partial least squares regression (PLSR), a back propagation neural network (BPNN) and a convolutional neural network (CNN) were used to construct the estimation models of SOM. The results indicated that the SNV could enhance the spectral information related to SOM and improve the accuracy of model estimation, and it was one of the most effective spectral pretreatment methods. Compared with the model constructed with the full-band spectroscopy method, the feature variable selection method could effectively improve the estimation accuracy of the Vis-NIR in situ spectroscopy model. The most obvious improvement was found with PSO, where R² and RPD were improved by more than 0.34 and 0.16, respectively, and RMSE was reduced by more than 0.29 g kg⁻¹. The accuracy of the CNN model was higher than that of the BPNN and PLSR models, both for the inversion model of SOM built from full-band spectral data and the bands selected by the characteristic variable selection method. SNV-PSO-CNN is the optimal hybrid model for in situ spectral measurement of SOM (R² = 0.71, RPD = 1.88, RMSE = 1.67 g kg⁻¹) and can realize the quantitative in situ spectral inversion of SOM in desert soils.