As a member of the death-associated protein kinase (DAPK) family, STK17B plays an important role in the regulation of cellular apoptosis and has been considered as a promising drug target for hepatocellular carcinoma. However, the highly conserved ATP-binding site of protein kinases represents a challenge to design selective inhibitors for a specific DAPK isoform. In this study, molecular docking, multiple large-scale molecular dynamics (MD) simulations, and binding free energy calculations were performed to decipher the molecular mechanism of the binding selectivity of PKIS43 toward STK17B against its high homology STK17A. MD simulations revealed that STK17A underwent a significant conformational arrangement of the activation loop compared to STK17B. The binding free energy predictions suggested that the driving force to control the binding selectivity of PKIS43 was derived from the difference in the protein–ligand electrostatic interactions. Furthermore, the per-residue free energy decomposition unveiled that the energy contribution from Arg41 at the phosphate-binding loop of STK17B was the determinant factor responsible for the binding specificity of PKIS43. This study may provide useful information for the rational design of novel and potent selective inhibitors toward STK17B.