Characteristics of random ocean waves have been measured by different devices, and X-band marine radar is one of the typical devices. This study proposes an enhanced methodology for estimating the significant wave height of ocean waves through the analysis of X-band radar images, particularly leveraging the shadowing characteristics inherent within radar images. The enhancement of the shadowing-based algorithm is achieved by incorporating three different key physical properties of ocean waves. These include the spatial autocorrelation function (SACF) in the Smith function, the orthogonal property of mean surface slopes, and the relationship of high-order spectral moments. The enhanced algorithm is complementarily integrated with fast Fourier transform (FFT)-based spectral analysis, facilitating the determination of significant wave height without the necessity for supplementary reference measurements. Numerical tests have been conducted using synthetic and real radar images corresponding to various sea states to validate the accuracy and reliability of the proposed methodology. The results demonstrate that the proposed techniques consistently improve the estimation accuracy of significant wave heights for both synthetic and real radar images. Even though the measured real radar images used for validation are not exhaustive in terms of the amount of dataset and range of sea state severity, considering that the proposed technique is in its early development stage, it is inspiring that its effectiveness and physical validity have been demonstrated through the present study.