The surrounding rock is in a complex stress environment and its mechanical behavior is also complex, especially after the excavation of the coal seam, the phenomenon of stress release of surrounding rock often occurs. The vertical stress and horizontal stress of the surrounding rock mass will have a series of complex changes. In underground engineering, rock mass is affected by dead weight pressure and tectonic stress. With coal mine production, the original stress of surrounding rock is demolished, and the destruction of surrounding rock is reflected in the loading and unloading failure of three-dimensional stress. Aiming at the phenomenon, this paper takes the Pingshuo East open-pit mine as the research background, and the experiments on physical and mechanical parameters of coal and rock mass was carried out, obtaining the coal and rock mechanics parameters, such as elastic modulus, Poisson’s ratio, internal friction angle, cohesive force, etc. The stress strain curve was created based on the conventional triaxial experiment of coal and rock under different confining pressure conditions. According to the characteristics of these curves, we obtain underground engineering rock mass unloading stress–strain variation characteristics. Through establishing a stress–strain equation based on confining pressure, we finally describe the mechanical failure characteristics of rock under triaxial stress.