As the central component of a pumped storage power station, the pump-turbine frequently enters the hump region during operation, which impacts the safe functioning of the unit. This study investigates the flow patterns at various guide vane openings within the hump area, employing the Omega vortex identification method. The results reveal that transitioning into the hump region from design conditions results in an unsteady flow state in the vaneless space, seriously threatening the stability of the pump-turbine. Notably, at the lowest point in the hump zone, the vaneless space generates high-speed circulation characterized by large velocity gradients and severe flow separation. The region of high turbulent kinetic energy intensifies with increasing guide vane opening. Additionally, low-frequency pressure pulsations become the predominant frequency in the hump zone, and the relative amplitude of these pulsations reach higher values, which is caused by rotating stall. The Omega vortex identification technique illustrates that the vortex structure is minimized during optimal efficiency in the vaneless space. As the flow rate in the hump region decreases, persistent high-energy vortices accumulate in the vaneless space, resulting in a significant increase in vorticity intensity and causing unsteady flow in the pump turbine.