The interaction of immune checkpoint molecules in the tumor microenvironment reduces the anti-tumor immune response by suppressing the recognition of T cells to tumor cells. Immune checkpoint inhibitor (ICI) therapy is emerging as a promising therapeutic option for cancer treatment. However, modulating the immune system with ICIs still faces obstacles with severe immunogenic side effects and a lack of response against many cancer types. Plant-derived natural compounds offer regulation on various signaling cascades and have been applied for the treatment of multiple diseases, including cancer. Accumulated evidence provides the possibility of efficacy of phytochemicals in combinational with other therapeutic agents of ICIs, effectively modulating immune checkpoint-related signaling molecules. Recently, several phytochemicals have been reported to show the modulatory effects of immune checkpoints in various cancers in in vivo or in vitro models. This review summarizes druggable immune checkpoints and their regulatory factors. In addition, phytochemicals that are capable of suppressing PD-1/PD-L1 binding, the best-studied target of ICI therapy, were comprehensively summarized and classified according to chemical structure subgroups. It may help extend further research on phytochemicals as candidates of combinational adjuvants. Future clinical trials may validate the synergetic effects of preclinically investigated phytochemicals with ICI therapy.