Ursodeoxycholic acid (UDCA) is a first-line clinical drug for the treatment of liver diseases. U12, a derivative of UDCA, showed effective anti-hepatoma activities in previous works. However, the low polarity and large doses limited the druglikeness of U12. In this study, the structural modification and optimization of U12 were further investigated and twelve U12 derivatives were synthesized by substitution, esterification and amidation reactions. The evaluation of the cytotoxicity of synthetic derivatives against hepatoma cell lines (HepG2) indicated that U12-I, U12a-d and U12h showed more effective cytotoxic effects on the growth of HepG2 cells than U12, and the preliminary structure–activity relationship was discussed. Among them, U12a exhibited the most potent anti-hepatocellular carcinoma activity. Mechanism studies indicated that U12a inhibited HepG2 cell proliferation by arresting the G0/G1 phase, and suppressed the activation of the PI3K/AKT/mTOR pathway. Further studies showed that U12a induced HepG2 cells apoptosis through activating the caspase signaling pathway. Furthermore, U12a evidently inhibits the growth of HepG2-derived tumor xenografts in vivo without observable adverse effects. Thus, U12a might be considered as a promising candidate for the treatment of hepatocellular carcinoma.