Infrared Small-Object Detection (IRSTD) presents a unique training challenge: objects typically span less than 0.01% of image pixels, so background gradients overwhelm the optimization process and compressed models tend to lose object information. It is found that this imbalance is significantly mitigated in the spectral domain—in the intermediate network layer, the gradient ratio of the target to the background reaches more than 30 times the observed value in the spatial domain. Building upon this observation, we present HiSGD (High–Low Spectral Gradient Decoupling), a principled framework that operates on local differential spectra rather than global feature maps. HiSGD introduces a Spectral Gradient Decoupling Operator (SGDO) that deterministically separates high-frequency target anomalies from low-frequency background stationarity via Differential Gaussian Decoupling (DGD)—a scale-aware local decomposition that matches the receptive field of typical IR small targets. The decoupled high-frequency band is supervised by an Adaptive Sparse-Edge Gradient Modulation (ASEGM) loss, which implicitly rebalances gradient contributions through an adaptive discrepancy sensitivity function without requiring explicit spatial masks. The low-frequency band is constrained by a Background Semantic Coherence Loss (BSCL) that enforces second-order statistical consistency rather than pixel-wise fidelity, preventing overfitting to sensor noise. By selectively distilling layers where spectral target contrast peaks, HiSGD enables lightweight student architectures that reduce parameters and FLOPs by 93% relative to their teachers while outperforming existing distillation methods on NUDT-SIRST and NUAA-SIRST. On the Jetson AGX Orin edge device, the distilled models reduce inference latency by 59.5% and 20.3%, validating the practical effectiveness of our approach for edge deployment.