In hyperspectral image (HSI) classification, each pixel is assigned to a specific land cover type, which is critical for applications in environmental monitoring, agriculture, and urban planning. Convolutional neural network (CNN) and Transformers have become widely adopted due to their exceptional feature extraction capabilities. However, the local receptive field of CNN limits their ability to capture global context, while Transformers, though effective in modeling long-range dependencies, introduce computational overhead. To address these challenges, we propose a frequency-domain and spatial-domain feature fusion network (FSFF-Net) for HSI classification, which reduces computational complexity while capturing global features. The FSFF-Net consists of a frequency-domain transformer (FDformer) and a deepwise convolution-based parallel encoder structure. The FDformer replaces the self-attention mechanism in traditional Visual Transformers with a three-step process: two-dimensional discrete Fourier transform (2D-DFT), adaptive filter, and two-dimensional inverse Fourier transform (2D-IDFT). 2D DFT and 2D-IDFT convert images between the spatial and frequency domains. Adaptive filter can adaptively retain important frequency components, remove redundant components, and assign weights to different frequency components. This module not only can reduce computational overhead by decreasing the number of parameters, but also mitigates the limitations of CNN by capturing complementary frequency-domain features, which enhance the spatial-domain features for improved classification. In parallel, deepwise convolution is employed to capture spatial-domain features. The network then integrates the frequency-domain features from FDformer and the spatial-domain features from deepwise convolution through a feature fusion module. The experimental results demonstrate that our method is efficient and robust for HSIs classification, achieving overall accuracies of 98.03%, 99.57%, 97.05%, and 98.40% on Indian Pines, Pavia University, Salinas, and Houston 2013 datasets, respectively.