Quantum dot (QD) photodetectors with low cost, a tunable spectral response range, the great possibility of multiple-exciton generation, and excellent detective performance have shown significant advantages in the fields of optical communication, imaging, and biological monitoring. However, QD photodetectors are facing challenges such as materials toxicity, poor stability, material integration difficulty, etc. Thus, the performance of these photodetectors still lags far behind their traditional Si-based counterparts, which have been widely applied commercially. This review presents a general overview focusing on the material selection and device architecture design, specifically on the evaluation parameters of the photodetector, the strategies to improve the quality of the QD active layer, the matching degree of the transport layer, and the efficiency of the constructed devices. Moreover, based on an in-depth fundamental understanding of the quantum size effect of various QDs, visible-, near-infrared, and short-wavelength infrared QD photodetectors are deliberated. The application practices and current research progress of QD photodetectors are summarized at the end, followed by potential improvements to inspire the development of QD photodetectors in more industries.