Hyperspectral image (HSI) classification remains a challenging task due to its high-dimensional spectral characteristics and the complex spatial heterogeneity of remote sensing scenes. Although significant progress has been made with convolutional neural networks (CNNs) and Transformer methods, existing approaches often fail to adequately model fine-grained variations in spectral curves, whilst struggling to strike a good balance between preserving local texture and modelling global context. To this end, this paper proposes a spatial–spectral dual-branch network (SFE-FM) for HSI classification, which combines spectral feature enhancement with a lightweight feature fusion mechanism to improve feature representational capacity. Specifically, a Spectral Feature Enhancement (SFE) module is designed to explicitly model spectral trends through first- and second-order differential operations, thereby enhancing potential discriminative information; at the same time, a lightweight Feature Mixing (FM) module is introduced into the network to model global dependencies across channels. In the dual-branch architecture, the spatial branch and the spectral branch extract spatial texture features and spectral semantic features respectively, and these features are adaptively reweighted using a cross-branch-guided fusion strategy (CSSF) to facilitate the effective integration of the two types of information. In addition, a multi-scale attention optimisation module (MSAO) has been introduced to enhance the response in key areas and improve the robustness of feature representations. Experiments conducted on the four benchmark datasets—WHU-Hi-HanChuan, Qingyun, Salinas and Pavia University, the proposed method achieved overall classification accuracies of 99.62%, 98.70%, 99.97% and 99.81%, respectively. Whilst maintaining a relatively low number of parameters (0.631M), it delivered competitive performance, demonstrating a good balance between classification accuracy and computational efficiency.