18 September 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
In recent years, 3D models have been utilized in many applications, such as auto-drivers, 3D reconstruction, VR, and AR. However, the scarcity of 3D model data does not meet its practical demands. Thus, generating high-quality 3D models efficiently from textual descriptions is a promising but challenging way to solve this problem. In this paper, inspired by the creative mechanisms of human imagination, which concretely supplement the target model from ambiguous descriptions built upon human experiential knowledge, we propose a novel text-3D generation model (T2TD). T2TD aims to generate the target model based on the textual description with the aid of experiential knowledge. Its target creation process simulates the imaginative mechanisms of human beings. In this process, we first introduce the text-3D knowledge graph to preserve the relationship between 3D models and textual semantic information, which provides related shapes like humans’ experiential information. Second, we propose an effective causal inference model to select useful feature information from these related shapes, which can remove the unrelated structure information and only retain solely the feature information strongly related to the textual description. Third, we adopt a novel multi-layer transformer structure to progressively fuse this strongly related structure information and textual information, compensating for the lack of structural information, and enhancing the final performance of the 3D generation model. The final experimental results demonstrate that our approach significantly improves 3D model generation quality and outperforms the SOTA methods on the text2shape datasets.
5 January 2023 in IEEE Access
In recent years, 3D model analysis has made a revolutionary development. Point cloud contains rich 3D object geometry information, which is an important 3D object data format widely used in many applications. However, the irregularity and disorder of the point cloud also cause its vulnerability to environmental impact, which may bring security risks to safety-critical 3D applications such as self-driving tasks. Recently, there are only a few methods engaged to attack the point cloud models to improve the robustness of point cloud analysis models. Most of them only focus on the attack by adjusting the points but ignore learning the perturbation’s distribution characteristics. In this work, we propose a novel framework to attack point cloud models. By introducing the GAN structure, we train a generator to produce slight point-to-point perturbations according to the sample’s raw classification, which can effectively boost the attack performance. Meanwhile, we propose an outlier removal module to constrain the magnitude of the generated perturbation. The goal is to guarantee the visual quality of generation samples to improve the difficulty of training and further improve the robustness of 3D analysis models. Finally, we carry out extensive attack experiments, and the related results demonstrate the effectiveness of our proposed method.
20 February 2025 in IEEE Transactions on Artificial Intelligence
Content-aware layout generation aims to produce fitting element arrangements based on the background contents, which is used for graphic design applications such as automatic poster layout design. In this paper, we propose ContentDM, a layout diffusion model specifically designed for the content-aware layout generation task, overcoming the limitations suffered from existing methods: irrational arrangement among layout elements and lack of refining ability for coarse generated results. ContentDM defines the layout diffusion process through random perturbations applied to both the position and type of layout elements. During the denoising training phase, the content-aware layout generator is trained to reconstruct samples from these perturbed layouts. This process enables the model to learn the correct arrangement patterns within the layout elements, thereby enhancing the rationality of generated layouts. Moreover, we develop an iterative layout inference strategy to enable the layout generator to refine the generated layouts progressively, thereby enhancing the overall quality of the generation results. Extensive experiments demonstrate that ContentDM significantly outperforms existing methods, achieving state-of-the-art performance in content-aware layout generation, both in terms of visual quality and quantitative metrics.
4 July 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Recently, advancements in text-to-image synthesis and image customization have drawn significant attention. Among these technologies, foreground-driven image synthesis models aim to create diverse scenes for specific foregrounds, showing broad application prospects. However, existing foreground-driven diffusion models struggle to accurately generate scenes with layouts that align with user intentions. To address these challenges, we propose CompCraft, a training-free framework that enhances layout control and improves overall generation quality in current models. First, CompCraft identifies that the failure of existing methods to achieve effective control arises from the excessive influence of fully denoised foreground information on the generated scene. To address this, we propose a foreground regularization strategy that modifies the foreground-related attention maps, reducing their impact and ensuring better integration of the foreground with the generated scene. Then, we propose a series of inference-time layout guidance strategies to guide the image generation process with the user’s finely customized layouts. These strategies enable current foreground-driven diffusion models with accurate layout control. Finally, we introduce a comprehensive benchmark to evaluate CompCraft. Both quantitative and qualitative results demonstrate that CompCraft can effectively generate high-quality images with precise customized layouts, showcasing its strong capabilities in pratical image synthesis applications.