Oil-immersed cooling provides an effective cooling scheme for high-power hub motors with compact structure and serious heating problems. However, with this cooling method, some oil friction loss will be generated, making the output torque and efficiency of the motor lower, which limits its application in the motor. It is essential to get an exact calculation of the oil friction loss so that it can be reduced in the future research. Firstly, a new method was proposed to improve the accuracy of oil friction loss calculated by an existing analytical method (Kori’s method), while the influence laws of oil-soaked depth and rotation speed on it were explored. Secondly, a three-dimensional transient Computational Fluid Dynamic (CFD) model based on Volume of Fluid (VOF) was established, considering the actual complex structure and the disorderly mixing of oil and air inside the motor. Finally, the oil friction loss calculated with an improved analytical method and a VOF model was verified by a testing. It was indicated that the VOF model was more precise but more time-consuming. The proposed method has the second highest accuracy but takes less time.