Magnetic anomaly detection (MAD) is used for detecting moving ferromagnetic targets. In this study, we present an end-to-end deep-learning model for magnetic anomaly detection on data recorded by a single static three-axis magnetometer. We incorporate an attention mechanism into our network to improve the detection capability of long time-series signals. Our model has good performance under the Gaussian colored noise with the power spectral density of $1/f^{\alpha }$, which is similar to the field magnetic noise. Our method does not require another magnetometer to eliminate the effects of the Earth’s magnetic field or external interferences. We evaluate the network’s performance through computer simulations and real-world experiments. The high detection performance and the single magnetometer implementation show great potential for real-time detection and edge computing.