The effects of climate change on agricultural systems raise important uncertainties about the future productivity and suitability of crops, especially in areas suffering from intense environmental changes. Olive groves occupy Mediterranean areas characterized by seasonal temporary droughts, which cause this cultivation to be highly dependent on local microclimatic conditions. Olive crop production can be reliably estimated using pollen intensity metrics together with post-pollination environmental conditions. In this study, we applied this kind of statistics-based models to identify the most relevant meteorological variables during the post-pollination periods for olive fruit production. Olive pollen time-series for the period of 1999–2012 was analyzed in 16 Italian provinces. Minimum and maximum temperature during spring and summer (March–August) showed a negative relationship with olive production, while precipitation always showed a positive correlation. The increase in aridity conditions observed in areas of Italy during the summer represents an important risk of decreasing olive crop production. The effect of climate change on the olive production trend is not clear because of the interactions between human and environmental factors, although some areas might show an increase in productivity in the near future under different climate change scenarios. However, as more drastic changes in temperature or precipitation take place, the risk to olive production will be considerably greater.