To address the synergistic stability evaluation of coal pillars and caving deposits in room-and-pillar mining of nearly vertical coal seams, this study takes the 101 Coal Mine (104th Regiment, Xishan Area, Urumqi, Xinjiang) as the engineering background. It combines physical similarity simulation and theoretical analysis to explore the synergistic bearing mechanism of coal pillars and caved deposits. Based on limit equilibrium theory, a combined instability criterion considering roof mudstone’s bending-toppling and shear-sliding is established; the Rankine earth pressure theory is modified, and a stability coefficient K_s (reflecting synergistic bearing effect) is proposed to realize quantitative evaluation of goaf stability. A model experiment simulates the mining of a 73° nearly vertical coal seam. Results show the roof instability mode (under coal pillars and caved deposits) is equivalent to anti-dip slope’s toppling-sliding composite failure. Experimental and theoretical results agree well, verifying the model’s rationality and applicability. This study provides a theoretical basis and analytical method for calculating the synergistic stability of coal pillars and caving deposits in nearly vertical coal seams.