Industrial agglomeration can promote economic growth through knowledge spillover and cooperation, while it may also bring serious pollution problems. Therefore, clarifying the relationship between industrial agglomeration and green development is of great significance to the realization of China’s sustainable development. In order to study the causal relationship between industrial agglomeration and green development, this paper first adopts the Overall Malquist Index method to more accurately measure the green development of Chinese cities characterized by environmental total factor productivity (ETFP). Subsequently, this paper uses the panel data of prefecture-level cities from 2003 to 2016 to study the causal relationship between industrial agglomeration and green development. We found that China’s current industrial agglomeration can promote green development. Mechanism analysis shows that industrial agglomeration can promote green development through improving the technological innovation, strengthening government intervention and optimizing the industrial structure. Finally, heterogeneity analysis shows that industrial agglomeration has a stronger role in promoting green development in areas where the economy is more developed, the degree of resource dependence is lower, and the degree of industrial agglomeration is higher. This paper not only provides theoretical and policy references for the research on industrial agglomeration and green development, but also provides experience reference for other developing countries.