The Xiaoai Jiedu recipe (XJR) is a traditional Chinese medicine formulation used in clinical settings to treat liver cancer. It has shown promising effectiveness by combining herbal and animal-derived ingredients, offering a new approach to cancer treatment. However, its mechanism of action is poorly understood. The molecular processes underlying the inhibitory effects of the XJR on hepatocellular cancer (HCC) were investigated. The primary chemical components of XJR and associated disease targets relevant to HCC were anticipated and compiled using a database. The potential targets and processes by which XJR influenced HCC were investigated using GO and KEGG enrichment analyses, as well as protein-protein interaction (PPI) networks. Transmission electron microscopy, laser confocal microscopy, and Western blotting were used to evaluate autophagy, while CCK-8 assays measured cell viability and Western blotting and flow cytometry evaluated apoptosis. In vivo assays were conducted employing an HCC xenograft mouse model. Network pharmacology analysis identified 456 intersecting targets between XJR and HCC. The top five active components are quercetin, cholesterol, jaceosidine, eupafolin, and oleanolic acid. The key targets include TP53, AKT1, IL6, EGFR, SRC, HSP90AA1, TNF, IL1B, MYC, and CASP3. Additionally, the autophagy pathway was found to be one of the main pathways through which XJR intervenes in HCC. The increased quantity of autophagosomes and autolysosomes, the overexpression of Beclin1 and LC3A/B-II proteins, and the downregulation of P62 all suggest that XJR stimulated autophagy in HCC cells. Functional tests employing pathway-specific activators and inhibitors and siRNA-based knockdown demonstrated that XJR promoted autophagy by blocking AKT/mTOR signaling. Furthermore, XJR reduced the viability of HCC cells and promoted apoptosis by upregulating apoptosis proteins. Autophagy inhibitors and Beclin1 silencing reversed these effects. Research conducted in vivo showed that XJR activated autophagy through the AKT/mTOR axis, thereby markedly reducing tumor growth and inducing tumor cell demise. These studies show that XJR activates autophagy in both cellular and animal models to induce apoptosis and decrease HCC cell proliferation, as shown by network pharmacology and verification assays. Further, these findings provide experimental evidence that the anti-tumor activity of XJR involves autophagy stimulation.