Based on the spectral reflection characteristics analysis of the muddy coastline in Jiangsu, an improved spectral water index (IWI) combined with the Otsu algorithm is proposed to extract muddy coastlines from Landsat Operational Land Imager (OLI) images. The IWI-extracted coastline results are compared with those extracted by the modified normalized difference water index (MNDWI), normalized difference water index (NDWI), enhanced water index (EWI), revised normalized different water index (RNDWI) and automated water extraction index (AWEI). The results show that the IWI is not affected by tidal conditions or sand content in the water, can reduce the “salt-and-pepper” phenomenon in the Otsu algorithm classification, can accurately identify water boundaries and can extract silty mudflats and marine buildings with high accuracy. It can also significantly increase the degree of automation of coastline extraction. The IWI combined with the Otsu algorithm demonstrates high accuracy of over 84% in the extraction muddy coastline data with one-pixel tolerance, which is twice as accurate as other indices. The accuracy of extraction for all other types of coastlines is over 81%. Therefore, the IWI index combined with the Otsu algorithm is reliable for studies of sea–land processes and coastline evolutions.