Hyperspectral image classification is widely applied in several fields. Since existing datasets focus on a single scene, current deep learning-based methods typically divide patches randomly on the same image as training and testing samples. This can result in similar spatial distributions of samples, which may incline the network to learn specific spatial distributions in pursuit of falsely high accuracy. In addition, the large variation between single-scene datasets has led to research in cross-scene hyperspectral classification, focusing on domain adaptation and domain generalization while neglecting the exploration of the generalizability of models to specific variables. This paper proposes two approaches to address these issues. The first approach is to train the model on the original image and then test it on the rotated dataset to simulate cross-scene evaluation. The second approach is constructing a new cross-scene dataset for spatial distribution variations, named GF14-C17&C16, to avoid the problems arising from the existing single-scene datasets. The image conditions in this dataset are basically the same, and only the land cover distribution is different. In response to the spatial distribution variations, this paper proposes a constrained spectral attention mechanism and a constrained spatial attention mechanism to limit the fitting of the model to specific feature distributions. Based on these, this paper also constructs a constrained spectral–spatial attention residual network (CSSARN). Extensive experimental results on two public hyperspectral datasets and the GF14-C17&C16 dataset have demonstrated that CSSARN is more effective than other methods in extracting cross-scene spectral and spatial features.