Hyperspectral video provides rich spatial and spectral information, which is crucial for object tracking in complex scenarios. Despite extensive research, existing methods often face an inherent trade-off between rich spectral information and redundant noisy information. This dilemma arises from the efficient utilization of hyperspectral image data channels. To alleviate this problem, this paper introduces a hierarchical spectral attention network for hyperspectral object tracking. We employ a spectral band attention mechanism with adaptive soft threshold to examine the correlations across spectral bands, which integrates the information available in various spectral bands and eliminates redundant information. Moreover, we integrate spectral attention into a hierarchical tracking network to improve the integration of spectral and spatial information. The experimental results on entire public hyperspectral competition dataset WHISPER2020 show the superior performance of our proposed method compared with that of several related methods in visual effects and objective evaluation.