When magnetic saturation occurs during the operation of a permanent magnet (PM) slotless brushless DC motor, the material permeability will no longer be a constant value, and the neglected magnetic saturation model used for motor optimization will no longer be applicable. And considering that the increase in motor torque will lead to a high temperature rise of the winding, therefore, an electromagnetic heat coupling model applicable to the occurrence of magnetic saturation in the motor is established, and the model is utilized in combination with the particle swarm algorithm to enhance the maximum output torque of the motor. Firstly, a 100 W, 16,400 r/min high-speed PM slotless DC brushless motor is taken as the object of study, and its electromagnetic–thermal coupling model is established to derive the analytical equations for the electromagnetic torque with respect to the split ratio, the thickness of the stator yoke, the PM thickness, and the copper loss. Secondly, based on the modeling, the motor was optimized using a particle swarm algorithm to maximize the output torque and minimize the copper loss. Finally, a prototype was fabricated and verified with the prototype through no-load and load experiments. The difference between the theoretical maximum output torque and the experimental maximum output torque is less than 8%. The results show that this method can effectively predict the maximum output torque of the motor in the case of magnetic saturation, and the model is suitable for increasing the maximum output torque of slotless brushless DC motors under space constraints.