Existing numerical modeling of three-dimensional (3D) slopes is performed mainly by using the shear strength reduction (SSR) technique based on the linear Mohr–Coulomb (MC) criterion, whereas the nonlinear failure criterion for rock slope stability is seldom used in slope modeling. However, it is known that rock mass strength is a nonlinear stress function and that, therefore, the linear MC criterion does not agree with the rock mass failure envelope very well. In this research, a nonlinear SSR technique is proposed that can use the Hoek–Brown (HB) criterion to represent the nonlinear behavior of a rock mass in the FLAC^3D program to analyze 3D slope stability. Extensive case studies are carried out to investigate the influence of the convergence criterion and boundary conditions on the 3D slope modeling. Results show that the convergence criterion used in the 3D model plays an important role, not only in terms of calculation of the factor of safety (FOS), but also in terms of the shape of the failure surface. The case studies also demonstrate that the value of the FOS for a given slope will be significantly influenced by the boundary condition when the slope angle is less than 50°.