The spatial characteristics of discontinuity traces play an important role in evaluations of the quality of rock masses. Most researchers have extracted discontinuity traces through the gray attributes of two-dimensional (2D) photo images or the geometric attributes of three-dimensional (3D) point clouds, while few researchers have paid attention to other important attributes of the original 3D point clouds, that is, the color attributes. By analyzing the color changes in a 3D point cloud, discontinuity traces in the smooth areas of a rock surface can be extracted, which cannot be obtained from the geometric attributes of the 3D point cloud. At the same time, a necessary filtering step has been designed to identify redundant shadow traces caused by sunlight on the rocks’ surface, and a multiscale spatial local binary pattern (MS-LBP) algorithm was proposed to eliminate the influence of shadows. Next, the geometric attributes of the 3D point cloud were fused to extract the potential discontinuity trace points on the rocks’ surface. For cases in which the potential discontinuity trace points are too scattered, a local line normalization thinning algorithm was proposed to refine the potential discontinuity trace points. Finally, an algorithm for establishing a two-way connection between a local vector buffer algorithm and a connectivity judgment algorithm was used to connect the discontinuity trace points to obtain the discontinuity traces of the rock mass’s surface. In addition, three datasets were used to compare the results extracted by existing methods. The results showed that the proposed method can extract the discontinuity traces of rock masses with higher accuracy, thereby providing data support for evaluations of the quality of rock masses and stability analyses.

The spatial variability in rock mass mechanical parameters critically affects slope stability assessments. This study investigated the southern slope of the Bayan Obo open-pit mine. A representative elementary volume (REV) with a side length of 14 m was determined through discrete fracture network (DFN) simulations. Based on the rock quality designation (RQD) data from 40 boreholes, a three-dimensional spatial distribution model of the RQD was constructed using Ordinary Kriging interpolation. The RQD values were converted into geological strength index (GSI) values through an empirical correlation, and the generalized Hoek–Brown criterion was applied to develop a spatially heterogeneous equivalent mechanical parameter field. Numerical simulations were performed using FLAC3D, with the slope stability evaluated using the point safety factor (PSF) method. For comparison, three homogeneous benchmark models based on the 5th, 25th, and 50th percentiles produced profile-scale safety factors of 0.96–1.92 and failed to replicate the observed failure geometry. By contrast, the heterogeneous model yielded safety factors of approximately 1.03–1.08 and accurately reproduced the mapped sliding surface. These findings demonstrate that incorporating spatial heterogeneity significantly improves the accuracy of slope stability assessments, providing a robust theoretical basis for targeted monitoring and reinforcement design.