We present a spectrum-based framework, utilizing low and high spectral latent features, to colorize the grayscale image. Current image colorization algorithms, despite significant achievements, constantly struggle with color bleeding and low color saturation. One of the main reasons causing such phenomena is vanilla convolutions in modern neural networks focus on local and fixed scales. To alleviate this issue, we propose a novel approach called Low and High Spectral Colorization(LoHiSC). LoHiSC is based on a new colorization network that uses fast fourier convolution(FFC), which is capable of efficiently connecting two distant locations and has a global receptive field. Particularly, our proposed LoHiSC improves FFC by splitting it into two main branches, one is cascaded with low pass spectral filter, and another with high pass spectral filter, for the reason we experimently find that colors are mainly concentrated in the low frequency part, while contours and edges in the high frequency part. Moreover, we propose a frequency domain loss which is beneficial to restore challenging artifacts of grayscale image colorization. Experiments on both synthetic dataset and real-world old photos demonstrate our method improves the stat-of-the-art and achieves compelling colorizing results.