Increasing evidence supported the possible neuro-invasion potential of SARS-CoV-2. However, no studies were conducted to explore the existence of the micro-structural changes in the central nervous system after infection. We aimed to identify the existence of potential brain micro-structural changes related to SARS-CoV-2. In this prospective study, diffusion tensor imaging (DTI) and 3D high-resolution T1WI sequences were acquired in 60 recovered COVID-19 patients (56.67% male; age: 44.10 ± 16.00) and 39 age- and sex-matched non-COVID-19 controls (56.41% male; age: 45.88 ± 13.90). Registered fractional anisotropy (FA), mean diffusivity (MD), axial diffusivity (AD), and radial diffusivity (RD) were quantified for DTI, and an index score system was introduced. Regional volumes derived from Voxel-based Morphometry (VBM) and DTI metrics were compared using analysis of covariance (ANCOVA). Two sample t-test and Spearman correlation were conducted to assess the relationships among imaging indices, index scores and clinical information. In this follow-up stage, neurological symptoms were presented in 55% COVID-19 patients. COVID-19 patients had statistically significantly higher bilateral gray matter volumes (GMV) in olfactory cortices, hippocampi, insulas, left Rolandic operculum, left Heschl's gyrus and right cingulate gyrus and a general decline of MD, AD, RD accompanied with an increase of FA in white matter, especially AD in the right CR, EC and SFF, and MD in SFF compared with non-COVID-19 volunteers (corrected p value <0.05). Global GMV, GMVs in left Rolandic operculum, right cingulate, bilateral hippocampi, left Heschl's gyrus, and Global MD of WM were found to correlate with memory loss (p value <0.05). GMVs in the right cingulate gyrus and left hippocampus were related to smell loss (p value <0.05). MD-GM score, global GMV, and GMV in right cingulate gyrus were correlated with LDH level (p value <0.05). Study findings revealed possible disruption to micro-structural and functional brain integrity in the recovery stages of COVID-19, suggesting the long-term consequences of SARS-CoV-2. Shanghai Natural Science Foundation, Youth Program of National Natural Science Foundation of China, Shanghai Sailing Program, Shanghai Science and Technology Development, Shanghai Municipal Science and Technology Major Project and ZJ Lab.