Mueller matrix ellipsometry (MME) is a powerful metrology tool for nanomanufacturing. The application of MME necessitates electromagnetic computations for inverse problems of metrology determination in both the conventional optimization process and the recent neutral network approach. In this study, we present an efficient, rigorous coupled-wave analysis (RCWA) simulation of multilayer nanostructures to quantify reflected waves, enabling the fast simulation of the corresponding Mueller matrix. Wave propagations in the component layers are characterized by local scattering matrices (s-matrices), which are efficiently computed and integrated into the global s-matrix of the structures to describe the optical responses. The performance of our work is demonstrated through three-dimensional (3D) multilayer nanohole structures in the practical case of industrial Muller matrix measurements of optical diffusers. Another case of plasmonic biosensing is also used to validate our work in simulating full optical responses. The results show significant numerical improvements for the examples, demonstrating the gain in using the RCWA method to address the metrological studies of multilayer nanodevices.