$1\times $ N optical switches are essential building blocks for modern information communication. Moreover, the recent development of a new on-chip solid-state light detection and ranging (Lidar) technology scheme has made this kind of switch attractive. In this work, we propose a new robust $1\times $ N optical switch structure that can get a 100 dB extinction ratio in theory by using two cascaded N $\times $ N optical diffractive units. As a proof-of-concept, a $1\times $ 5 optical switch is demonstrated, and a more than 40 dB extinction ratio for all channels is achieved at the center wavelength of 1550 nm. The ultra-high extinction ratio and large fabrication tolerance of our structure will pave a way for future information communication and Lidar.