Oil-in-water nanoemulsions (NE) can increase the water solubility of plant-derived bioactive molecules as drug candidates. Machine learning-guided NE design can prevent the expensive, time-consuming trial-and-error process. NE composition data was aggregated into a dataset; a predictive machine learning model identified improved self-nanoemulsifying system formulations (olive oil and combinations of Tween 20, Tween 80, glycerol, and soy lecithin). Predictive power was assessed by estimating successful self-nanoemulsification through transmittance and Dynamic Light Scattering. NEs were loaded with an organic extract containing anacardic acid. Encapsulation efficiency was measured by UHPLC. Antiproliferative activity was evaluated on human hepatic cancer (Hep G2) and normal-like human embryonic kidney (HEK-293) cell lines. The model showed an accuracy of 81%. The best-performing formulation, consisting of 10% olive oil, 60% Tween 20, and 30% glycerol, exhibited an average particle size of 162.8 ± 26 nm, a polydispersity index of 0.234 ± 0.03, and high encapsulation efficiency. While HEK-293 cells remained unaffected, naked NE exhibited a selective growth inhibitory effect on the Hep G2 cell line. Loaded NE increased the cytotoxic effect on Hep G2 (IC50: 5.9 ± 1.27 µM). Machine learning-guided NE formulation was a successful carrier for the plant extract and the molecule of interest, providing a proof of concept for how artificial intelligence can shorten the development pipeline for NE drug delivery systems.