The yield criterion of rock is significant for the identification of the whole failure process and the stability analysis of rock engineering. Studies of yield criteria from energy aspects are critical but rare. This paper investigated the yield criterion using strain energy and a modified compounded mobilized planes (CMP) model. The yield strain energy and strength criterion were formulated with different combinations of stress variables. Based on conventional triaxial and true triaxial tests, the evolution laws of mechanical parameters, including elastic parameters and strength parameter, were quantitatively determined. Accordingly, a new parameter correlation model, intermediate principal stress effect, and prerequisite for convex yield surface were investigated with theoretical and experimental approaches, respectively. With respect to previous criteria, the studied criterion has some advantages. Firstly, parameter correlations can be characterized with this criterion. Secondly, the shape of the yield surface is criterion-dependent and relies on rock properties, and can be applied to wide range of rocks. Thirdly, the estimation of yield stress is quasi-independent of the sample variability, which resulted in good agreement with test data. Fourthly, the criterion can be transformed into some classic criteria widely used in civil engineering. The criterion in this study offers an alternative viewpoint. In addition, it has high calculation precision and wide application for varied rocklike geomaterials.