Commercial poly methyl methacrylate (PMMA)-based cement is currently used in the field of orthopedics. However, it suffers from lack of bioactivity, mechanical weakness, and monomer toxicity. In this study, a PMMA-based cement nanocomposite reinforced with hydroxyapatite (HA) nanofibers and two-dimensional (2D) magnesium phosphate MgP nanosheets was synthesized and optimized in terms of mechanical property and cytocompatibility. The HA nanofibers and the MgP nanosheets were synthesized using a hydrothermal homogeneous precipitation method and tuning the crystallization of the sodium-magnesium-phosphate ternary system, respectively. Compressive strength and MTT assay tests were conducted to evaluate the mechanical property and the cytocompatibility of the PMMA-HA-MgP nanocomposites prepared at different ratios of HA and MgP. To optimize the developed nanocomposites, the standard response surface methodology (RSM) design known as the central composite design (CCD) was employed. Two regression models generated by CCD were analyzed and compared with the experimental results, and good agreement was observed. Statistical analysis revealed the significance of both factors, namely, the HA nanofibers and the MgP nanosheets, in improving the compressive strength and cell viability of the PMMA-MgP-HA nanocomposite. Finally, it was demonstrated that the HA nanofibers of 7.5% wt and the MgP nanosheets of 6.12% wt result in the PMMA-HA-MgP nanocomposite with the optimum compressive strength and cell viability.