Grazing and cultivation are two important management practices worldwide that can cause significant soil organic carbon (SOC) losses. However, it remains elusive how soil microbes have responded to soil carbon changes under these two practices. Based on a four-decade long field experiment, this study investigated the effects of grazing and cultivation on SOC stocks and microbial properties in the semi-arid grasslands of China. We hypothesize that grazing and cultivation would deplete SOC and depress microbial activities under both practices. However, our hypotheses were only partially supported. As compared with the adjacent indigenous grasslands, SOC and microbial biomass carbon (MBC) were decreased by 20% or more under grazing and cultivation, which is consistent with the reduction of fungi abundance by 40% and 71%, respectively. The abundance of bacteria and actinomycetes was decreased under grazing but increased under cultivation, which likely enhanced microbial diversity in cultivation. Invertase activity decreased under the two treatments, while urease activity increased under grazing. These results suggest that nitrogen fertilizer input during cultivation may preferentially favor bacterial growth, in spite of SOC loss, due to rapid decomposition, while overgrazing may deteriorate the nitrogen supply to belowground microbes, thus stimulating the microbial production of nitrogen acquisition enzymes. This decade-long study demonstrated differential soil microbial responses under grazing and cultivation and has important applications for better management practices in the grassland ecosystem.