Remote sensing image change detection refers to the automatic acquisition of the location and category of object changes using images of different phases. The rapid development of optical remote sensing and artificial intelligence technology has made deep learning a mainstream method for remote sensing change detection. In current research, deep learning-based remote sensing change detection mainly focuses on determining whether changes have occurred at the pixel level, and this pixel-level change detection is susceptible to pseudo-changes due to factors such as differences in imaging angles of optical cameras, changes in imaging seasons, and occlusion. In addition, in some specific scenarios, such as the change analysis task for movable objects with regular geometries (e.g., airplane, vehicles, etc.) is more concerned with the change of the object as a whole than the change of the pixels inside the object, in which case pixel-level methods are no longer the best choice and object-level change detection methods should be adopted. To address the above problems, we propose spatial-frequency difference attention network for object-level change detection (SFDA-OCDet). Firstly, we use a Siamese backbone with shared weight parameters to extract the image features of different stages from the input dual-temporal images. Secondly, we design the spatial difference attention (SDA) module, which obtains the attention weights from the local context and the global context to enhance the learning ability of the difference features of different time images from the spatial domain. Then, we propose the wavelet frequency difference attention (WFDA) module, which utilizes the discrete wavelet transform and the attention mechanism to enhance the difference feature learning capability from the frequency domain. Finally, we present the wavelet balanced feature pyramid (WBFP) module, which introduces the discrete wavelet transform and its inverse transformation to implement the downsampling and upsampling process to avoid feature information loss. The change detection head outputs change detection results in the form of a bounding box. We construct four object-level change detection datasets, Airplane-CD, Vehicle-CD, Building-CD and Wind turbine-CD, to carry out experimental validation. Numerous experiments have shown that the proposed SFDA-OCDet outperforms other comparative methods, proving the validity of the method.