The literature has mixed findings regarding the impact of climate change on agricultural production, probably due to different choices of dependent variables. Based on panel data analysis of 140 countries from an extended period, i.e., 1961 to 2013, this study applies the explicit method of using TFP as the dependent variable, but also delivers estimations with the implicit approach of using agricultural output as the dependent variable, so as to be able to directly compare the results with the mainstream trends in the existing literature. We found that using agricultural TFP as the dependent variable generates more consistent and robust results. We also found a strong negative impact of climate change on agricultural productivity. A one-unit downward deviation of precipitation (i.e., drought) and one unit of upward deviation of temperature (i.e., global warming) decrease the TFP by 0.530 and 0.494, respectively. As we have one of the largest sample sizes when it comes to studying the effect of climate change, we carried out an exploration by dividing the sample into different categories of developed vs. developing countries and cold vs. warm climates, as well as by splitting the time series into two distinct periods. We found that climate change has had a heterogeneous impact on countries with different development levels, with negative impacts on developing countries and positive effects on developed countries, which leads to the rising concern of the impact of climate change on inequality. This heterogeneity and the uneven impact found in this study strongly suggest the need for countries to work together to coordinate and mitigate these adverse effects rather than to adapt to the situation by focusing on the input variations, as the latter will increase the inequality effect of climate change and add to an already unstable global environment.