Low-carbon city construction (LCC) is an important strategy for countries desiring to improve environmental quality, realize cleaner production, and achieve sustainable development. Low-carbon cities have attracted widespread attention for their attempts to coordinate the relationship between environmental protection and economic development. Using the panel data from 2006 to 2017 of prefecture-level cities in China, this study applied the difference-in-differences (DID) method to analyze the effects of LCC on the total factor productivity (TFP) of the cities and its possible transmission mechanism. The results show significantly positive effects on TFP, but the effects on each component of TFP are different. Although the LCC has promoted technical progress and scale efficiency, it has inhibited technical efficiency. The accuracy of the results has been confirmed by several robustness tests. Mechanism analysis showed that the pilot policy of low-carbon cities has promoted technical progress and scale efficiency by technological innovation and the upgrading of industrial structure, but resource mismatches among enterprises have been the main reason for reduced technical efficiency. Regional heterogeneity analysis showed that the effects on TFP in the eastern region have been more significant than in the central and western regions. In the eastern region, they have promoted technical progress, while in the central and western regions, they have promoted technical progress and scale efficiency but hindered technical efficiency. This paper presents our findings for the effects of LCC on economic development and provides insightful policy implications for the improvement of technical efficiency in low-carbon cities.