The nonreciprocal propagation of light at the single-photon level is essential for building a quantum network. Bulk optical schemes are lossy and difficulty to integrate onto a chip. We propose a single-photon optical diode and a three-port circulator without a magnetic field by coupling an unbalanced quantum impurity to a passive, linear optical waveguide or a whispering-gallery-mode microresonator which supports a locally or globally circularly polarized photon. Thanks to the unbalanced quantum Jaynes-Cummings coupling, the optical nonreciprocal propagation of single photons can be achieved without an external magnetic field. In particular, the three-port single-photon circulator can be accomplished using the existing experimental technology. The optical isolation can be reversed by selectively populating the initial state of the quantum impurity. Moreover, by using an ensemble of identical atoms filled in a hollow-core microbottle resonator, nonreciprocal propagation of weak light pulses can be achieved.