Cropland non-grain and non-agriculturalization monitoring (CNNM) is of great significance for ensuring national food security. Recently, semantic segmentation has been a promising approach that can fully exploit the advantages of high-resolution remote sensing imagery. It is of great significance to clarify the performance of semantic segmentation algorithms for CNNM and their influencing factors. This paper contributes to research on deep learning-based semantic segmentation models for CNNM by investigating the robustness and generalization ability of detection models in complex scenarios. To investigate these models, seven mainstream segmentation models were examined with respect to two self-constructed unmanned aerial vehicle (UAV)-based cropland monitoring datasets under various experimental settings. The experimental results reveal several key findings: For the datasets employed in the present study, remote sensing semantic segmentation models demonstrate superior performance relative to generic visual models. Increasing backbone depth does not guarantee significant performance improvements. The recognition challenges posed by task-specific categories such as agricultural facility land further underscore the need for tailored deep learning semantic segmentation algorithms for CNNM. These experimental results provide valuable insights into the practical performance of deep learning semantic segmentation models and offer useful guidance for future semantic segmentation research in the field of CNNM.