Human and organizational factors are the contributing factors for collision accidents from the historical data. To discover the key influencing factor, a human factor analysis and classification system based Bayesian Network model is proposed in this paper. The kernel of this proposed model is first to derive the unsafe acts from the perspective of perception, decision-making, and execution failures using the collision avoidance scheme, to classify the human factors into five categories using the modified human-factor analysis and classification system, and to transform the influencing factors of HOFs in the modified HFACS into the graphical structure of the Bayesian network. The results are verified from historical collision accidents data in the Yangtze River, and sensitivity analysis is carried out to validate the axioms of the Bayesian network. From further analysis, the causation factor and global causation chain of ship collision accidents can be derived. Consequently, the results are beneficial for the prevention and control of ship collision accidents in the Yangtze River by reducing human and organization factors.