Effective abnormal behavior detection in ship operations is essential for ensuring navigational safety and operational efficiency in marina ports. This study presents a hybrid method that integrates statistical analysis and neural network modeling to detect abnormal behavior based on data obtained through leisure boat sea trials. Detection criteria were established based on ship motion characteristics, operating area conditions, and the properties of the sea trial data. The method combines Rayda’s criterion and standard deviation thresholds to identify sudden changes in measured data, while a Long Short-Term Memory (LSTM) network is used to predict normal ship behavior. Deviations between predicted and measured values were evaluated using three thresholds (levels 1, 2, and 3), with level 3 effectively isolating the most significant abnormal data (representing 2–10% of the data). The proposed method is capable of successfully identifying sudden acceleration or deceleration, unusual course changes, extended stationary periods, deviations from expected routes, complex maneuvers, and track continuity issues. The results demonstrate that the proposed hybrid method can reliably distinguish abnormal ship behaviors based on real sea trial data. To separate true abnormalities from false alarms or sensor and environmental noise, its practical application on a real ship is planned as future work. This study provides a foundation for intelligent ship monitoring systems and supports the development of autonomous and semi-autonomous navigation technologies.