Existing hyperspectral anomaly detection methods primarily focus on spectral or spatial differences between anomalies and the background. However, these methods often overlook spectral aliasing around anomalous targets and the interference of anomalous pixels in background reconstruction. To address these issues, an adaptive Taylor decomposition and polynomial kernel differential fusion network is proposed. First and foremost, the adaptive Taylor decomposition method is proposed to perform spectral decomposition of the hyperspectral image. Moreover, the decomposed hyperspectral image is fed into a polynomial kernel differential fusion module to enhance multi-scale feature processing capabilities. Then, a weakened diagonal convolution module is constructed to reduce the influence of diagonal pixels on the tested pixels. At last, the residual is computed to obtain the final anomaly detection results. The key advantage of this method lies in enhancing the distinction between the background and anomalous targets, while effectively mitigating the degradation of background reconstruction quality caused by anomalies. Experimental results show that the method improves background reconstruction and network robustness. The proposed method also outperforms ten state-of-the-art methods.