In recent years, convolutional neural networks (CNNs) have become a potent tool for hyperspectral image classification (HSIC), where classification accuracy, computational cost, and generalization ability are the main focuses. In this study, a novel approach to hyperspectral image classification is proposed. A tandem spectral–spatial attention module (TAM) was designed to select significant spectral and spatial features automatically. At the same time, a dense pyramidal residual module (DPRM) with three residual units (RUs) was constructed, where feature maps exhibit linear growth; a dense connection structure was employed between each RU, and a TAM was embedded in each RU. Dilated convolution structures were used in the last two layers of the pyramid network, which can enhance the network’s perception of fine textures and features, improving information transfer efficiency. Tests on four public datasets, namely, the Pavia University, Salinas, TeaFarm, and WHU-Hi-HongHu datasets, were carried out, and the classification accuracies of our method were 99.60%, 99.95%, 99.81%, and 99.84%, respectively. Moreover, the method enhanced the processing speed, especially for large datasets such as WHU-Hi-HongHu. The training time and testing time of one epoch were 53 s and 1.28 s, respectively. Comparative experiments with five methods showed the correctness and high efficiency of our method.