17 February 2023 in ACM Transactions on Multimedia Computing, Communications, and Applications
Recent works on single image high dynamic range (HDR) reconstruction fail to hallucinate plausible textures, resulting in information missing and artifacts in large-scale under/over-exposed regions. In this article, a decoupled kernel prediction network is proposed to infer an HDR image from a low dynamic range (LDR) image. Specifically, we first adopt a simple module to generate a preliminary result, which can precisely estimate well-exposed HDR regions. Meanwhile, an encoder-decoder backbone network with a soft mask guidance module is presented to predict pixel-wise kernels, which is further convolved with the preliminary result to obtain the final HDR output. Instead of traditional kernels, our predicted kernels are decoupled along the spatial and channel dimensions. The advantages of our method are threefold at least. First, our model is guided by the soft mask so that it can focus on the most relevant information for under/over-exposed regions. Second, pixel-wise kernels are able to adaptively solve the different degradations for differently exposed regions. Third, decoupled kernels can avoid information redundancy across channels and reduce the solution space of our model. Thus, our method is able to hallucinate fine details in the under/over-exposed regions and renders visually pleasing results. Extensive experiments demonstrate that our model outperforms state-of-the-art ones.
22 August 2023 in Displays
20 April 2026 in IEEE Transactions on Circuits and Systems for Video Technology
In recent years, integrating multimodal information, particularly visual and echo data, has shown great promise for improving depth estimation performance. While existing works demonstrate that combining binaural echo features with image attributes can enhance depth estimation, they often use rudimentary feature alignment and fusion methods, failing to fully exploit the complementary nature of cross-modal information and limiting integration effectiveness. To address these challenges, this paper introduces an innovative multimodal fusion framework. First, the framework incorporates a combination of multi-scale self-attention and cross-attention mechanisms, establishing correlations between features and facilitating cohesive interactions between the visual and echo domains. Furthermore, we propose an incremental feature updating mechanism based on Convolutional Gated Recurrent Units (ConvGRU), which implements cascaded iterative optimization, integrating contextual features with the multi-scale fused features from both echo and image modalities. In each iteration, the framework preserves contextual information from previous steps while employing a multi-level loss function to guide result updates. This approach effectively captures spatial structural information and progressively enhances depth estimation accuracy. Comprehensive experimental evaluations on the Replica, Matterport3D and BatVision (BV1) datasets validate the effectiveness of the proposed method. Comparative analyses with state-of-the-art monocular plus echo methods underscore the superior performance achievable through this novel framework.