Magnetic inversion through three-dimensional (3D) susceptibility reconstruction can effectively identify the deep extension characteristics and structural variations in faults. Therefore, the reliability of inversion results from magnetic anomaly data is a key issue that must be addressed in fault detection and quantitative evaluation of fault activity. In recent years, deep neural network-driven magnetic data inversion methods have rapidly become a research focus in the field of geophysical magnetic data inversion. However, existing methods primarily rely on convolutional neural networks (CNNs), whose inherent local feature extraction capabilities limit their ability to model the spatial continuity of large-scale subsurface magnetic structures. Moreover, the general lack of prior physical constraints in these network models often leads to unreliable inversion results. To address these limitations, this paper proposes a physics-informed multi-scale deep learning inversion method for magnetic anomaly data. The method designs a dual-stream Transformer-CNN fusion module (TCFM). It leverages the self-attention mechanism in Transformers to model global susceptibility correlations while efficiently capturing local geological features through CNN convolutional operations. This enables collaborative modeling of multi-scale subsurface magnetic structures, significantly enhancing inversion accuracy. Furthermore, by incorporating deep physical priors, we design a depth-aware weighted loss function. By strengthening optimization constraints in deep regions, it effectively improves the vertical resolution of inversion models for deep magnetic structures. Comparative experiments with U-Net++ and Transformer demonstrate that the proposed method achieves smaller errors and higher inversion accuracy. Applied to measured aeromagnetic data from the Dandong region of China, the method yields reliable inversion results. Variations in magnetic susceptibility within these results successfully delineate the spatial distribution of fault zones, providing a geophysical basis for regional seismic hazard monitoring and assessment.