The improved phase congruency (PC) algorithms have been successfully applied to optical and synthetic aperture radar (SAR) image registration since they are insensitive to nonlinear radiometric and geometric differences. However, most of the algorithms are sensitive to large-scale differences and rotation differences between optical and SAR images. To tackle this, we propose a PC framework to register optical and SAR images. It is compatible with large-scale and rotation invariance. Firstly, a multi-scale Harris keypoint extraction method based on the maximum moment of PC (named PC-Harris) is proposed. The scale space is constructed by combining PC with the log-Gabor filter. Secondly, we propose a PC model to construct the feature descriptors. The orientation and amplitude responses are obtained based on the PC model. Meanwhile, the novel descriptor is constructed based on the polar coordinate system and thus can handle the scale and rotation differences between optical and SAR images. Finally, outliers are removed by the fast sample consensus (FSC). The experiments conducted on several optical and SAR images verify the effectiveness of the proposed framework.