Underwater image enhancement (UIE) technology plays a vital role in marine resource exploration, environmental monitoring, and underwater archaeology. However, due to the absorption and scattering of light in underwater environments, images often suffer from blurred details, color distortion, and low contrast, which seriously affect the usability of underwater images. To address the above limitations, a lightweight transformer-based model (LITM) is proposed for improving underwater degraded images. Firstly, our proposed method utilizes a lightweight RGB transformer enhancer (LRTE) that uses efficient channel attention blocks to capture local detail features in the RGB domain. Subsequently, a lightweight HSV transformer encoder (LHTE) is utilized to extract global brightness, color, and saturation from the hue–saturation–value (HSV) domain. Finally, we propose a multi-modal integration block (MMIB) to effectively fuse enhanced information from the RGB and HSV pathways, as well as the input image. Our proposed LITM method significantly outperforms state-of-the-art methods, achieving a peak signal-to-noise ratio (PSNR) of 26.70 and a structural similarity index (SSIM) of 0.9405 on the LSUI dataset. Furthermore, the designed method also exhibits good generality and adaptability on unpaired datasets.