Ultra-large underground metal mines often have complex surrounding rock structures, making traditional assessment methods inadequate for warning against the sudden failure of highly brittle rock masses. To accurately identify high-risk rock layers, this study combines Brazilian splitting tests with acoustic emission (AE) monitoring on four typical surrounding rocks. A normalized damage–stress brittleness coefficient (NDBC) is proposed, and Gaussian mixture model (GMM) clustering is employed to analyze crack evolution mechanisms. Different from conventional brittleness indexes merely based on mechanical parameters, the proposed NDBC characterizes rock brittleness from the perspective of progressive damage evolution driven by acoustic emission microfracture information, providing a dynamic evaluation basis for sudden instability in highly brittle rock masses. The GMM clustering automatically identifies crack features and accurately quantifies the transition from tensile peak to increasing shear during the failure process. The research shows that: (1) AE characteristics during the failure stage are manifested as medium- to high-frequency signals caused by small-scale cracks. (2) Siliceous limestone exhibits extremely high brittleness (NDBC of 0.07) and sudden failure due to the difficulty of microcrack propagation, posing a greater risk of instability and potential overall collapse during mining; in contrast, granite (NDBC of 0.23) is more ductile, showing progressive damage accumulation. (3) Initial rock splitting failure is primarily tensile cracking, with shear cracking increasing as failure approaches, transitioning the failure mechanism to a tensile–shear composite mode. Therefore, establishing a differentiated monitoring and prevention system based on AE main frequency identification and GMM analysis, designating siliceous limestone surrounding rock areas as key prevention zones, can effectively reduce the risk of sudden instability and ensure safe mining operations.