A high-speed advancing ship will cause significant squats in shallow water, which could increase the risk of grounding. To this end, a program based on the Rankine higher-order boundary element method (HOBEM) is developed to investigate a high-speed displacement ship with a transom stern moving in shallow water. The nonlinear free surface condition is satisfied by adopting an iterative algorithm on the real free surface. The transom condition is considered by implementing a modified transom condition. Computations of wave-making resistance, sinkage and trim in deep water are first performed, and satisfactory agreement is achieved by comparing with the experimental results; the simulations are then extended to the shallow water case. It indicates that the present method can provide a suitable balance of practicability and robustness, which can be considered as an efficient tool for the guidance in ship design stage.