In remote sensing image scene classification (RSISC) tasks, downsampling is crucial for reducing computational complexity and cache demands, enhancing the model’s generalization capability of deep neural networks. Traditional methods, such as regular fixed lattice approaches (pooling in CNN and token merging in transformers), often flatten distinguishing texture features, impacting classification performance. To address this, we propose an adaptive token merging transformer (ATMformer) that preserves essential local features by estimating the importance score of each token. This allows significant tokens to be isolated during merging, mitigating the risk of feature blurring. Our experiments on three widely used RSISC datasets (NWPU-RESISC45, Aerial Image Dataset, and EuroSAT) demonstrate that ATMformer achieves state-of-the-art performance across all datasets. These findings underscore the importance of effective downsampling techniques in maintaining classification accuracy while efficiently processing large-scale data.