21 July 2023 in IEEE Transactions on Visualization and Computer Graphics
Recent neural rendering methods have made great progress in generating photorealistic human avatars. However, these methods are generally conditioned only on low-dimensional driving signals (e.g., body poses), which are insufficient to encode the complete appearance of a clothed human. Hence they fail to generate faithful details. To address this problem, we exploit driving view images (e.g., in telepresence systems) as additional inputs. We propose a novel neural rendering pipeline, Hybrid Volumetric-Textural Rendering (HVTR++), which synthesizes 3D human avatars from arbitrary driving poses and views while staying faithful to appearance details efficiently and at high quality. First, we learn to encode the driving signals of pose and view image on a dense UV manifold of the human body surface and extract UV-aligned features, preserving the structure of a skeleton-based parametric model. To handle complicated motions (e.g., self-occlusions), we then leverage the UV-aligned features to construct a 3D volumetric representation based on a dynamic neural radiance field. While this allows us to represent 3D geometry with changing topology, volumetric rendering is computationally heavy. Hence we employ only a rough volumetric representation using a pose- and image-conditioned downsampled neural radiance field (PID-NeRF), which we can render efficiently at low resolutions. In addition, we learn 2D textural features that are fused with rendered volumetric features in image space. The key advantage of our approach is that we can then convert the fused features into a high-resolution, high-quality avatar by a fast GAN-based textural renderer. We demonstrate that hybrid rendering enables HVTR++ to handle complicated motions, render high-quality avatars under user-controlled poses/shapes, and most importantly, be efficient at inference time. Our experimental results also demonstrate state-of-the-art quantitative results.
1 August 2022 in IEEE Transactions on Visualization and Computer Graphics
In this paper, we propose a controllable high-quality free viewpoint video generation method based on the motion graph and neural radiance fields (NeRF). Different from existing pose-driven NeRF or time/structure conditioned NeRF works, we propose to first construct a directed motion graph of the captured sequence. Such a sequence-motion-parameterization strategy not only enables flexible pose control for free viewpoint video rendering but also avoids redundant calculation of similar poses and thus improves the overall reconstruction efficiency. Moreover, to support body shape control without losing the realistic free viewpoint rendering performance, we improve the vanilla NeRF by combining explicit surface deformation and implicit neural scene representations. Specifically, we train a local surface-guided NeRF for each valid frame on the motion graph, and the volumetric rendering was only performed in the local space around the real surface, thus enabling plausible shape control ability. As far as we know, our method is the first method that supports both realistic free viewpoint video reconstruction and motion graph-based user-guided motion traversal. The results and comparisons further demonstrate the effectiveness of the proposed method.
18 September 2023 in IEEE Geoscience and Remote Sensing Letters
Infrared (IR) small dim target detection near the sea-sky line (SSL) is crucial for enhancing the early warning capability of maritime vehicles. However, the interferences caused by the strong contrast have not been properly addressed. Consequently, a specially designed algorithm Regional-Division Local Contrast Measure (RDLCM) that focuses on the detection of infrared small dim targets appearing near the SSL is proposed. First, an SSL detection module based on a lightweight Convolutional Neural Network (CNN) is devised to achieve fast pixel-level SSL detection. Then, a set of regional-division windows (RDW) are designed according to the strong grayscale contrast distribution around the SSL, through the division of the effective regions, the RDWs could realize the potential extraction and refinement of the IR small dim targets that appear near the SSL. Experiments on three IR image sequences demonstrate that the proposed algorithm achieves the best detection accuracy among the classical and the state-of-the-art algorithms in comparison, and runs at 44 frames per second (FPS) which could meet real-time requirements. The code and dataset are available at RDLCM.
10 April 2025 in IEEE Transactions on Instrumentation and Measurement
Infrared (IR) ship detection in unmanned aerial vehicles (UAV) imagery under maritime environments encounters challenges due to the complexity of sea-sky backgrounds. Current research efforts have not fully leveraged the unique characteristics of maritime scenes to distinguish this task from general IR target detection. To overcome these limitations, we propose a Two-Stage Sea-Sky Region-Segmentation-Guided IR Ship Enhancement and Detection (TIRSD) learning network, specifically designed for targeted ship enhancement and detection in maritime environments. The process is structured into two main stages: enhancement and detection. In the enhancement stage, pixel-level sea-sky region segmentation is accomplished via sea-sky line (SSL) detection, which is followed by precise IR ship enhancement and interference suppression, utilizing specially designed morphological filters. For the detection stage, the Bilateral Hierarchical Target Encoding Module (BHTEM) is introduced to encode both high- and low-dimensional features, ensuring optimal detail retention and extraction. Additionally, the Iterative Fusion Enhancement Midlayer (IFEM) with neighborhood reinforcement operations is presented. Furthermore, a newly controllable Local Discrete Attention Mechanism (LDAM) is utilized to focus on small ships, thereby improving detection efficiency. TIRSD surpasses the current leading methods in terms of accuracy, achieving a Mean Average Precision (mAP) of 76.9% on the proposed dataset and 52.4% on a public dataset. Comprehensive experimental results underscore TIRSD’s effective utilization of the unique characteristics of maritime environments, indicating promising application prospects.
30 November 2023 in IEEE Geoscience and Remote Sensing Letters
Fast and accurate infrared (IR) sea–sky line (SSL) detection could greatly benefit the efficiency of target detection under maritime. However, the traditional SSL detection algorithms are slightly inferior in detection accuracy, while the convolutional neural network (CNN)-based algorithms have high standard hardware and dataset requirements, which are difficult to satisfy in some practical scenes. In this article, a novel concise and intuitive SSL detection algorithm named bisection window and homogeneity principle-based local contrast measure (BHLCM) is proposed. First, a bisection local contrast window (BLCW) is proposed based on the local contrast measure (LCM) and be used for the search of the patches that contain SSL segments along a set of variable preset vertical paths. Then, based on the analysis of the physical characteristics of areas near SSL, the homogeneity principle is proposed to remove the patches containing false SSL segments. Finally, the midpoints of the remaining SSL segments are used to fit the final SSL through random sample consensus (RANSAC). Experimental results based on four IR image sequences with more than 3000 IR images illustrate that compared with the state-of-the-art algorithms, BHLCM not only achieves precision comparable to the most accurate one but also is significantly ahead in speed among most of the algorithms compared. In addition, speed tests about BHLCM based on different hardware platforms show that real-time detection could still be achieved even without high-performance graphics cards. The code and dataset are available at BHLCM https://github.com/FJsRepo/BHLCM .