As ship traffic volumes increase and navigable waters become more complex, vessel traffic service operators (VTSOs) face growing challenges to effectively monitor marine traffic. To address the heavy reliance on human expertise in current ship supervision, we propose a method for quickly identifying high risk ships to enhance the situational awareness of VTSOs in complex waters. First, the K-means clustering algorithm is improved using the Whale Optimization Algorithm (WOA) to adaptively cluster ships within a waterway, segmenting the traffic in the area into multiple ship clusters. Second, a ship cluster collision risk assessment model is developed to quantify the degree of collision risk for each ship cluster. Finally, a weighted directed complex network is constructed to identify high risk ships within each ship cluster. Experimental simulations show that the proposed WOA–K-means clustering algorithm outperforms other adaptive clustering algorithms in terms of computation speed and accuracy. The developed ship cluster collision risk assessment model can identify high risk ship clusters that require VTSO attention, and the weighted directed complex network model accurately identifies high risk ships. This approach can assist VTSOs in executing a comprehensive and targeted monitoring process encompassing macro, meso, and micro aspects, thus boosting the efficacy of ship oversight, and mitigating traffic hazards.