Mechanical system reliability analysis constitutes a primary research focus in the field of engineering. This study aims to address the issue of complex mechanical systems with intricate mechanisms and nonlinear reliability equations that are challenging to solve. To this end, we present a reliability analysis and optimization methodology that merges the response surface and sensitivity analysis methods. A comprehensive formation of reliability assessment and optimization of complex mechanical systems is achieved by creating a response surface model to fit the complex state function and solving the reliability parameters, followed by an error sensitivity analysis to determine the mechanical system’s reliability adjustment strategy. Finally, these methods are applied to a cylindrical material transport device to preliminarily realize the reliability assessment and average reliability optimization goals. The study’s findings may offer a theoretical framework and research opportunities to evaluate and enhance the reliability of intricate mechanical systems.