Accurate exploitation of spatial structures and spectral characteristics is essential for fine-grained crop classification using remote sensing imagery. Although multi-source remote sensing data provide complementary information, most existing methods implicitly assume homogeneous data sources with consistent spatial resolution. In practice, high spatial resolution and rich spectral information are usually provided by different sensors, making cross-source spatial–spectral fusion a non-trivial challenge. To address this issue, we propose SSF-TransUnet, a dual-branch spatial–spectral joint modeling framework for fine crop classification. The proposed network explicitly decouples spatial structure extraction and spectral discriminability learning by jointly utilizing high spatial resolution imagery and multi-spectral observations acquired from different satellite sensors within a unified architecture. To support model training and evaluation, we construct SSCR-Agri, a spatial–spectral complementary resolution agricultural dataset integrating meter-level GF-2 imagery and multi-spectral Sentinel-2 data from five representative agricultural regions in northern China, covering five crop categories including corn, rice, wheat, potato, and others. Extensive experiments demonstrate that SSF-TransUnet consistently outperforms representative CNN-based and hybrid CNN–Transformer models. The proposed method achieves an overall accuracy (OA) of 81.84% and a mean Intersection over Union (mIoU) of 0.6954 in fine-grained crop classification, effectively distinguishing crops. These results highlight the effectiveness of spatial–spectral joint modeling for high-resolution crop mapping and demonstrate its potential for precision agriculture and large-scale agricultural monitoring applications, and shows a promising mechanism when combined with multi-temporal observations.