Src homology 2 domain-containing protein tyrosine phosphatase (SHP2) is a non-receptor protein tyrosine phosphatase encoded by the Ptpn11 gene, which regulates cell growth, differentiation and apoptosis via modulating various signaling pathways, such as the RAS/ERK signaling pathway, and participates in the PD-1/PD-L1 pathway governing immune surveillance. It has been recognized as a breakthrough antitumor therapeutic target. Besides, numerous studies have shown that SHP2 plays an important role in the regulation of inflammatory diseases. However, inhibitors targeting the active site of SHP2 lack drug-likeness due to their low selectivity and poor bioavailability, thus none has advanced to clinical development. Recently, allosteric inhibitors that stabilize the inactive conformation of SHP2 have achieved breakthrough progress, providing the clinical proof for the druggability of SHP2 as an antitumor drug target. This paper reviews the recently reported design and discovery of SHP2 small molecule inhibitors, focused on the structure–activity relationship (SAR) analysis of several representative SHP2 inhibitors, outlining the evolution and therapeutic potential of the small molecule inhibitors targeting SHP2.