Currently, electromagnetic radiation and interference have a significant effect on the operation of electronic devices and human health systems. Thus, developing excellent microwave absorbers have a huge significance in the material research field. Herein, a kind of ultrafine zinc oxide (ZnO) nanoparticles (NPs) supported on three-dimensional (3D) ordered mesoporous carbon spheres (ZnO/OMCS) is prepared from silica inverse opal by using phenolic resol precursor as carbon source. The prepared lightweight ZnO/OMCS nanocomposites exhibit 3D ordered carbon sphere array and highly dispersed ultrafine ZnO NPs on the mesoporous cell walls of carbon spheres. ZnO/OMCS-30 shows microwave absorbing ability with a strong absorption (− 39.3 dB at 10.4 GHz with a small thickness of 2 mm) and a broad effective absorption bandwidth (9.1 GHz). The outstanding microwave absorbing ability benefits to the well-dispersed ultrafine ZnO NPs and the 3D ordered mesoporous carbon spheres structure. This work opened up a unique way for developing lightweight and high-efficient carbon-based microwave absorbing materials.