In this study, the updated Global Inventory Modeling and Mapping Studies (GIMMS) Normalized Difference Vegetation Index (NDVI) dataset for growing season (April to October), which can better reflect the vegetation vigor, was used to investigate the interannual variations in NDVI and its relationship with climatic factors, in order to preliminarily understand the climate impact on vegetation and provide theoretical basis for the response of ecosystem to climate change. Multivariate linear regression models, including the Ordinary Least Squares (OLS) and Geographically Weighted Regression (GWR), were adopted to analyze the correlation between NDVI and climatic factors (temperature and precipitation) together. Average growing-season NDVI significantly increased at a rate of 0.0015/year from 1982 to 2013, larger than several regions in China. On the whole, its relationship with temperature is positive and also stronger than precipitation, which indicated that temperature may be a limiting factor for the vegetation growth in the Karst region. Moreover, the correlation coefficients between grassland NDVI and climatic factors are the largest. Under the background of NDVI increasing trend from 1982 to 2013, the period of 2009–2012 was chosen to investigate the influencing factors of a sharp decline in NDVI. It can be found that the reduced temperature and solar radiation, caused by the increase in cloud cover and precipitation, may play important roles in the vegetation cover change. All in all, the systematic research on the interannual variations of growing-season NDVI and its relationship with climate revealed the heterogeneity and variability in the complicated climate change in the Karst ecosystem for the study area. It is the Karst characteristics that hinder obtaining more representative conclusions and tendencies in this region. Hence, more attention should be paid to promoting Karst research in the future.