Clarifying the relationship between urbanization and farmland abandonment in urban agglomerations (UAs) is crucial to guide the formulation of arable land management policies and strategies for sustainable urban development. Despite numerous studies confirming the correlation between farmland abandonment and certain urbanization factors, the exploration of the patterns and underlying mechanisms of farmland abandonment in China’s UAs remains worthy of systematic investigation. In this study, we conducted an analysis of the spatiotemporal trends in farmland abandonment and examined the key drivers of farmland abandonment in four representative Chinese UAs—Beijing–Tianjin–Hebei (BTH), Chengdu–Chongqing (CC), Pearl River Delta (PRD), and Yangtze River Delta (YRD). Our findings reveal that farmland abandonment has been intensified with increasing fragmentation and aggregation patches across these UAs. Abandonment experience was the main driver of continuous abandonment. Moreover, natural conditions persistently influenced farmland abandonment in the BTH, while land urbanization and economic urbanization were predominant drivers in the CC. The abandonment in the PRD was mainly driven by population urbanization, while the abandonment in the YRD was primarily driven by economic urbanization and land urbanization. The research findings provide data support and scientific explanation for land policy-making in these typical UAs under different development strategies.