With the development of high-resolution Synthetic Aperture Radar (SAR) systems, researchers are increasingly paying attention to the application of SAR offset tracking methods in ground deformation estimation. The traditional normalized cross correlation (NCC) tracking method is based on regular matching windows. For areas with different moving characteristics, especially the landslide boundary areas, the NCC method will produce incorrect results. This is because in landslide boundary areas, the pixels of the regular matching window include two or more types of moving characteristics: some pixels with large displacement, and others with small or no displacement. These two kinds of pixels are uncorrelated, which result in inaccurate estimations. This paper proposes a new offset tracking method with SAR images based on the adaptive matching window to improve the accuracy of landslide displacement estimation. The proposed method generates an adaptive matching window that only contains pixels with similar moving characteristics. Three SAR images acquired by the Jet Propulsion Laboratory’s Uninhabited Aerial Vehicle Synthetic Aperture Radar (UAVSAR) system are selected to estimate the surface deformation of the Slumgullion landslide located in the southwestern Colorado, USA. The results show that the proposed method has higher accuracy than the traditional NCC method, especially in landslide boundary areas. Furthermore, it can obtain more detailed displacement information in landslide boundary areas.