China’s Chang’e lunar exploration project obtains digital orthophoto image (DOM) and digital elevation model (DEM) data covering the whole Moon, which are critical to lunar research. The DOM data have three resolutions (i.e., 7, 20 and 50 m), while the DEM has two resolutions (i.e., 20 and 50 m). Analysis and research on these image data effectively help humans to understand the Moon. In addition, impact craters are considered the most basic feature of the Moon’s surface. Statistics regarding the size and distribution of impact craters are essential for lunar geology. In existing works, however, the lunar surface has been reconstructed less accurately, and there is insufficient semantic information regarding the craters. In order to build a three-dimensional (3D) model of the Moon with crater information using Chang‘e data in the Chang‘e reference frame, we propose a four-step framework. First, software is implemented to annotate the lunar impact craters from Chang’e data by complying with our existing study on an auxiliary annotation method and open-source software LabelMe. Second, auxiliary annotation software is adopted to annotate six segments in the Chang’e data for an overall 25,250 impact crater targets. The existing but inaccurate craters are combined with our labeled data to generate a larger dataset of craters. This data set is analyzed and compared with the common detection data. Third, deep learning detection methods are employed to detect impact craters. To address the problem attributed to the resolution of Chang’e data being too high, a quadtree decomposition is conducted. Lastly, a geographic information system is used to map the DEM data to 3D space and annotate the semantic information of the impact craters. In brief, a 3D model of the Moon with crater information is implemented based on Chang’e data in the Chang‘e reference frame, which is of high significance.