Deep learning-based methods have shown promising performance in CSI-based fingerprint localization. However, current approaches often adopt feature interaction across different base stations (i.e., channels) throughout the entire process to achieve base station modeling. This approach conflates interactions between antennas within the same base station and those between different base stations, leading to inefficiencies. Additionally, processing each CSI point as a separate unit introduces a significant computational burden. In this work, we propose a new neural network, called HG-Net, which can use single-channel features to obtain a more generalized learning representation and achieve better localization performance. Specifically, HG-Net adopts a Patchify for coarse-grained feature extraction. Then, Single Channel Feature Extractor extracts feature from a single channel, while Hierarchical Feature Aggregation stores three semantic features of different granularity and merges them to obtain rich deep features. In addition, a Recurrent Neural Network is developed to obtain long-range frequency domain dependencies. Finally, the linear layer-based Positioner can map the deep features to locations. Experimental results on three real-world datasets show that our proposed HG-Net achieves an average positioning deviation of 0.14 m, which greatly exceeds the state-of-the-art approaches.