The intertwined charge, spin, orbital, and lattice degrees of freedom could endow 5d compounds with exotic properties. Current interest is focused on electromagnetic interactions in these materials, whereas the important role of lattice geometry remains to be fully recognized. For this sake, we investigate pressure-induced phase transitions in the spin-orbit Mott insulator Sr₃Ir₂O₇ with Raman, electrical resistance, and x-ray diffraction measurements. We reveal an interesting magnetic transition coinciding with a structural transition at 14.4 GPa, but without a concurrent insulator-metal transition. The conventional correlation between magnetic and Mott insulating states is thereby absent. The observed softening of the one-magnon mode can be explained by a reduced tetragonal distortion, while the actual magnetic transition is associated with tilting of the IrO₆ octahedra. This work highlights the critical role of lattice frustration in determining the high-pressure phases of Sr₃Ir₂O₇. The ability to control electromagnetic properties via manipulating the crystal structure with pressure promises a new way to explore new quantum states in spin-orbit Mott insulators.