Water and water transfer are the keys of the concrete durability problem; the non-uniform moisture transfer caused by the concrete micro–meso structure has a great effect on the drying shrinkage crack, transfers of inimical ions, etc. For the non-uniform moisture transfer problem, a multi-scale concrete moisture diffusion coefficient model which can consider the effect of Knudsen diffusion was established and verified based on the moisture transfer mechanism of porous medium and the concrete micro–meso structure characteristics. The effects of pore structure, the interfacial transition zone, and aggregate on the concrete moisture diffusion coefficient were studied based on the model, and the non-uniform moisture transfer characteristics and differences in concrete wetting and drying were analyzed via simulations. The results show that the moisture transfers more easily via the pores ranging from 10 nm to 100 nm, the effect of Knudsen diffusion increases with the increasing water-to-cement ratio and decreases with the increasing relative moisture, and Knudsen diffusion is also an effect factor which causes the moisture diffusion coefficient to increase with the increase in moisture. Moisture transfers more easily via the interfacial transition zone at the meso-level and causes a “flow around” phenomenon. The “S” growth relation between the moisture diffusion coefficient and relative moisture can consider the differences in the moisture diffusion coefficient under wetting and drying conditions to a certain extent, which makes concrete wet faster than dry. In addition, the jumping growth of the moisture diffusion coefficient in the relation also leads to an “inflection point” in the concrete moisture distribution.