Optimizing the design of Ceramic Composite Armor Systems (CCASs) is crucial for liberating redundant design space and enhancing ballistic performance. However, existing solution frameworks still exhibit deficiencies in handling multi-constrained optimization, particularly regarding surrogate model generalization and the nonlinearity and non-convexity of solution algorithms, leading to suboptimal accuracy and efficiency. This study proposes a novel Response Surface Methodology–Adaptive Moment Estimation (RSM-Adam) framework to address these challenges. The developed RSM surrogate model integrates key CCAS failure characteristics, such as structural deformation, fiber/matrix failure and interfacial debonding/delamination into the optimization analysis using few finite element analysis (FEA). This approach overcomes the limitation of conventional theoretical models that focus solely on ballistic performance, establishing a foundation for a generalized multi-constrained optimization model. Meanwhile, an improved Adam method incorporating a cosine step-size adjustment strategy demonstrates superior capability in handling nonlinearity and non-convexity during iteration compared to conventional algorithms like the Simulated Annealing Algorithm (SAA), Particle Swarm Optimization (PSO), and Genetic Algorithm (GA). It achieves over 60% reduction in computational error compared with conventional intelligent algorithms, while maintaining a computational cost comparable to or lower than most of them, proving highly effective and efficient for complex multi-constrained models. The rationality and accuracy of the RSM-Adam framework are validated through ballistic testing and FEA of an Al₂O₃/Carbon Fiber (CF)/Aramid Fiber (AF) composite armor sample, with systematic analysis of the influence of key framework parameters on optimization outcomes. It is also confirmed that the proposed RSM-Adam framework provides a practical, widely applicable, and engineering-valuable solution for the multi-constrained optimal design of the CCAS.