Phage display-mediated immuno-polymerase chain reaction (PD-IPCR) is an ultrasensitive detection technology that combines the advantages of immuno-PCR and phage display. The phage particle, which displayed antibody fragments including single-chain fragment variable (scFv), variable domain of heavy-chain antibodies (VHH), and antigen-binding fragment (Fab) on the surface can be directly used in IPCR, supplying both the detection antibody and deoxyribonucleic acid (DNA) template. In this work, we used ochratoxin A (OTA) as a model system to study the capacity of PD-IPCR in the detection of toxic small molecular weight compounds, especially mycotoxins. An alpaca-derived VHH library was constructed and subjected to four cycles of panning. In total, 16 clones with four unique sequences were selected by competitive binding with OTA. The clone VHH-28 resulted in the lowest 50% inhibitory concentration of 0.31 ng/mL in the phage enzyme-linked immunosorbent assay (ELISA) and was selected to develop the VHH phage-based real-time immuno-PCR (RT-IPCR). The detection limit of the VHH phage-based RT-IPCR was 3.7 pg/L, with a linear range of 0.01–1000 pg/mL. This method was compared with conventional ELISA, and validation results indicated the reliability of VHH phage-based RT-IPCR in the detection of OTA in cereal samples. This study provides a new idea for the ultrasensitive detection of mycotoxins and other toxic small molecular weight compounds.