In recent years, three-dimensional (3D) point clouds, which are applicable in various fields such as the metaverse and immersive communication, are attracting increasing attention. Under constrained storage and bandwidth conditions, efficient point cloud compression (PCC) plays a crucial role. To address these challenges, the Moving Picture Experts Group has been actively developing the geometry-based point cloud compression (G-PCC) standard and has recently proposed a test model for dynamic solid point clouds called Solid G-PCC. However, several issues still hinder the coding efficiency of attributes, such as inaccurate prediction, redundant coding bits, and accumulated distortion due to dependencies between frames. To tackle these challenges, we propose a layer-based rate-distortion optimized (RDO) attribute coding (L-RDOAC) method. This approach incorporates a layer-based RDO prediction (L-RDOP) to enhance prediction accuracy, a layer-based RDO quantization (L-RDOQ) to minimize redundant coding bits, and a layer-based RDO Wiener filter (L-RDOWF) to reduce distortion. Experimental results demonstrate that the coding efficiency of the proposed method significantly outperforms the state-of-the-art G-PCC reference software, as assessed through both objective and subjective evaluations. Specifically, compared to the state-of-the-art GeS-TM version 7.0, the proposed L-RDOAC achieves average Bjøntegaard-delta (BD) rates of -7.94%, -10.95%, and -8.16% for Luma, Cr, and Cb, respectively, under the C1 configuration (lossless geometry with lossy attributes), while under the C2 configuration (lossy geometry with lossy attributes), the average BD-rates are -7.88%, -8.09%, and -4.87%, respectively, when octree-based geometry coding is used.