An oil in water (O/W) nanoemulsion formulation containing kojic monooleate (KMO) in thin film system was developed. Response surface methodology (RSM) was used to optimize and analyzed the effect of three variables, namely concentration of polyvinyl alcohol (PVA) (20–30% w/w), concentration of propylene glycol (PG) (1–10% w/w), and shear rate of high shear homogenizer (3000–9000 rpm) on droplet size as a response, while other compositions remained constant such as KMO (10.0% w/w), Tween 80 (3.19% w/w), castor oil (3.74% w/w), xanthan gum (0.70% w/w), and germall plus (0.7% w/w, PG (and) diazolidinyl urea (and) iodopropynyl butylcarbamate). The optimized KMO nanoemulsion formulation with desirable criteria was PVA (27.61% w/w) and PG (1.05% w/w), and shear rate (8656.17 rpm) with a predicted droplet size (110.21 nm) and actual droplet size (105.93 nm) with a residual standard error (RSE) of less than 2.0% was obtained. Analysis of variance (ANOVA) showed that the fitness of the quadratic polynomial fit the experimental data with a F-value of 65.30, p–value of p < 0.0001, and a non-significant lack-of-fit. The optimized KMO formulation shows the desired criteria of the thin film system and the physicochemical properties (Zeta potential −37.37 mV, PDI 0.13, pH 4.74) and stability at four different conditions indicate its suitability for cosmeceutical applications.