For photonic applications, the inverse design method plays a critical role in the optimized design of photonic devices. According to its two ingredients, inverse design in photonics can be improved from two aspects: to find solutions to Maxwell’s equations more efficiently and to employ a more suitable optimization scheme. Various optimization algorithms have been employed to handle the optimization: the adjoint method (AM) has become the one of the most widely utilized ones because of its low computational cost. With the rapid development of deep learning (DL) in recent years, inverse design has also benefited from DL algorithms, leading to a new pattern of photon inverse design. Unlike the AM, DL can be an efficient solver of Maxwell’s equations, as well as a nice optimizer, or even both, in inverse design. In this review, we discuss the development of the AM and DL algorithms in inverse design, and the advancements, advantages, and disadvantages of the AM and DL algorithms in photon inverse design.