This study focuses on a marine two-stage tandem hybrid planetary system. Natural frequencies and vibration modes are determined using a translational–torsional coupled dynamic model. Based on the motion characteristics of the transmission system, free vibration is categorized into three typical modes. The parameter sensitivity of natural frequencies is computed, and the effects of structural parameters such as unequally spaced planet, mesh stiffness, planet mass and rotational inertia on the natural frequencies are analyzed. Utilizing the coupling factor, the mode transition criterion for the natural frequencies response to parameters is formulated. The results demonstrate that the vibration modes of the two-stage tandem hybrid planetary system can be classified as the fixed-axis train vibration mode, the differential train vibration mode, and the coupled vibration mode. Unequally spaced planet primarily disrupts vibration modes without significantly affecting natural frequencies. In contrast, the effects of mesh stiffness, planet mass and rotational inertia on the modes are opposite to those of unequally spaced planets. The validity of the parameter sensitivity and mode transition criterion is substantiated through illustrative examples.