Image-based human age estimation has been a hot topic in the field of face attribute recognition. As the greatest dilemma of age estimation is the similar appearance of adjacent age labels, this paper presents a two-stage age estimation model based on the idea of coarse-to-fine decision strategy and fine-grained feature learning mechanism. In the first stage, Swin Transformer is employed to perform global age estimation for each image, and then an adaptive age range is obtained by local adjusted the global age estimation result. This stage gives a relative age trend of the input image. In the second stage, a local regressor is performed on the generated age range based on a sub-network combining semantic attention mechanism and hierarchical fine-grained feature pooling. The proposed semantic attention provides a guidance of age-sensitive regions for distinguishing fine-grained age-relevant feature in the second stage. The formulation of the proposed two-stage framework is intuitive and end-to-end trainable, and experiments on three popular age estimation benchmarks achieve the state-of-the-art results.