Protein phosphatases, as the counterpart to protein kinases, are essential for homeostatic balance of cell signaling. Small chemical compounds that modulate the specific activity of phosphatases can be powerful tools to elucidate the biological functions of these enzymes. More importantly, many phosphatases are central players in the development of pathological pathways where inactivation can reverse or delay the onset of human diseases. Therefore, potent inhibitors for such phosphatases can be of great therapeutic benefit. In contrast to the seemingly identical enzymatic mechanism and structural characterization of eukaryotic protein kinases, protein phosphatases evolved from diverse ancestors, resulting in different domain architectures, reaction mechanisms and active site properties. In this review, we discuss for each family of serine/threonine protein phosphatases their involvement in biological processes and corresponding strategies for small chemical intervention. Recent advances in modern drug discovery technologies have markedly facilitated the identification of selective inhibitors for some members of the phosphatase family. Furthermore, the rapid growth in knowledge about structure–activity relationships related to possible new drug targets has aided the discovery of natural product inhibitors for the phosphatase family. This review summarizes the current state of investigation of the small molecules that regulate the function of serine/threonine phosphatases, the challenges presented and also strategies to overcome these obstacles.