Rapid urbanization has exerted immense pressure on urban environments, severely constraining the growth of ancient trees. The growth of ancient trees is closely linked to the microbial communities in their rhizospheres, and studying their community characteristics may provide new insights into promoting the growth and rejuvenation of ancient trees. In this study, the rhizosphere soil and root systems of ancient Ginkgo biloba trees (approximately 200 years old) and adult G. biloba trees (approximately 50 years old) in Shanghai were selected as research subjects. Phospholipid fatty acid (PLFA) analysis and high-throughput sequencing were employed to investigate the diversity of microbial communities in the G. biloba rhizosphere. The results indicated that the 19 PLFA species selected to characterize the soil microbial community structure and biomass were present in the rhizosphere soil of both ancient and adult G. biloba trees. However, the total microbial biomass and the microbial biomass in the rhizosphere soil of ancient G. biloba were lower than the microbial biomass in the rhizosphere soil of adult G. biloba. The biomasses of Gram-negative bacteria (G⁻), arbuscular mycorrhizal fungi (AMF), and protozoans (P) were significantly different. Total phosphorus, organic matter, and pH may be the key factors influencing the soil microbial community in the rhizosphere zone of ancient G. biloba. An in-depth study of AMF showed that the roots and rhizosphere soil of G. biloba contained abundant AMF resources, which were assigned to 224 virtual taxa using the MaarjAM reference database, belonging to four orders, ten families, and nineteen genera. The first and second most dominant genera were Glomus and Paraglomus, respectively. Archaeospora and Ambispora were more dominant in the rhizosphere than the roots. Furthermore, the abundance of live AMF was significantly higher in ancient G. biloba than in adult G. biloba. Therefore, future research should focus on the improvement of soil environmental characteristics and the identification and cultivation of indigenous dominant AMF in the rhizosphere of ancient G. biloba, aiming for their effective application in the rejuvenation of ancient trees.