Image-based virtual try-on (VTON) systems based on deep learning have attracted research and commercial interests. Although they show their strengths in blending the person and try-on clothing image and synthesizing the dis-occluded regions, their results for complex-posed persons are often unsatisfactory due to the limitations in their geometry deformation and texture-preserving capacity. To address these challenges, we propose CloTH-VTON+ for seamlessly integrating the image-based deep learning methods and the strength of the 3D model in shape deformation. Specifically, a fully automatic pipeline is developed for 3D clothing model reconstruction and deformation using a reference human model: first, the try-on clothing is matched to the target clothing regions in the simple shaped reference human model, and then the 3D clothing model is reconstructed. The reconstructed 3D clothing model can generate a very natural pose and shape transfer, retaining the textures of clothes. A clothing refinement network further refines the alignment, eliminating the misalignment due to the errors in human pose estimation and 3D deformation. The deformed clothing images are combined utilizing conditional generative networks to in-paint the dis-occluded areas and blend them all. Experiments on an existing benchmark dataset demonstrate that CloTH-VTON+ generates higher quality results in comparison to the state-of-the-art VTON systems and CloTH-VTON. CloTH-VTON+ can be incorporated into extended applications such as multi-pose guided and Video VTON.

Recently, image-based virtual try-on (VTON) systems using deep generative models have drawn significant research attention. However, the 2D clothing shape transform methods in the earlier works show serious limitations in 3D clothing deformation required in multiple-pose VTON scenarios. In this paper, we develop a 3D-MPVTON system of two pipelines and show that a 3D clothing model reconstruction approach provides much better results for the multi-pose VTON scenario. First, the 3D clothing model reconstruction pipeline is based on CloTH-VTON+. The try-on clothing is matched to the target clothing regions in the simply-shaped reference human model and its 3D model is reconstructed using the associated 3D human body model. For natural clothing rendering from an arbitrary view, accurate texture mapping is crucial. We have developed a highly accurate texture matching method. The try-on pipeline first generates target segmentation from the target pose for conditional information for the following stage's network models. Correct target segmentation was one of the main performance bottlenecks in previous VTON studies. Our proposed equalized entropy loss for the target segmentation generation network greatly reduces the segmentation label imbalance, and results in high-quality segmentation and reduced training time. The rigged reconstructed 3D clothing model can be easily deformed into the target pose and the human body shape while retaining the textures of the clothes. The remaining parts, i.e., the non-target clothing regions of the human in the target pose are generated through a deep generative human pose transfer model. Finally, the generated clothing and the remaining parts are combined using conditional generative networks to in-paint the dis-occluded areas and blend them together. Our proposed hybrid pipeline outperforms the previous 2D-based approaches by substantial margins in both objective evaluation and subjective user study, especially in the cases of the large pose and viewpoint changes.

The shape and pose estimation of a human body are essential for human behavior analysis, sports and medical analysis, and virtual reality. Although 2D image data are much easier to acquire than 3D scan data, the estimation accuracy using 2D images is still far below the 3D scanning methods’ ones. In this paper, we propose a 2D image-based human body estimation method appropriate for body shape and size measurement. The proposed method uses SMPL (Skinned Multi-Person Linear model), a human body model, and rich correspondences of Dense-Pose network so named Dense-Pose2SMPL, which estimates the SMPL parameters estimation by minimizing the re-projection error of the correspondences between the pixels in the human image and 3D surface points in SMPL. The previous SMPL parameter estimation methods use the sparse joint correspondence so that they show very limited performance in shape estimation. We compare the body measurement accuracy of Dense-Pose2SMPL with a single human image with SMPLify, a joint-based estimation method, and DecoMR, a Dense-Pose-based neural network regression method. The experiments results show a dramatic improvement of Dense-Pose2SMP over SMPLify and DecoMR. In the circumference estimation error, over 30 percent decrease for the overweight and underweight and 10 to 20 percent decrease on average. We also provide an analysis of the effects of the input conditions: the subject’s BMI level, pose, clothing style, and camera viewpoint. A-pose, side (profile) camera views and minimal and tight clothing style show higher accuracy than other conditions. Finally, the extension with multiple (two) images can give around 15 percent more improvement in body size estimation accuracy over the single image-based method.