Dewei Zhou

Dewei Zhou

Auto-generated profile, based on publicly available information

Dewei Zhou

Zhejiang University, Hangzhou, China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 June 2024
Verified
Conference Paper
Closed access
DZDewei Zhou
Dewei Zhou
YLYou Li
You Li
FMFan Ma
Fan Ma
XZXiaoting Zhang
Xiaoting Zhang
YYYi Yang
Yi Yang
We present a Multi-Instance Generation (MIG) task, si-multaneously generating multiple instances with diverse controls in one image. Given a set of predefined coordinates and their corresponding descriptions, the task is to ensure that generated instances are accurately at the designated locations and that all instances' attributes adhere to their corresponding description. This broadens the scope of current research on Single-instance generation, elevating it to a more versatile and practical dimension. Inspired by the idea of divide and conquer, we introduce an innovative approach named Multi-Instance Generation Controller (MIGC) to address the challenges of the MIG task. Ini-tially, we break down the MIG task into several subtasks, each involving the shading of a single instance. To ensure precise shading for each instance, we introduce an instance enhancement attention mechanism. Lastly, we aggregate all the shaded instances to provide the necessary information for accurately generating multiple instances in stable diffusion (SD). To evaluate how well generation models per-form on the MIG task, we provide a COCO-MIG bench-mark along with an evaluation pipeline. Extensive experiments were conducted on the proposed COCO-MIG bench-mark, as well as on various commonly used benchmarks. The evaluation results illustrate the exceptional control ca-pabilities of our model in terms of quantity, position, at-tribute, and interaction. Code and demos will be released at https://migcproject.github.io/.

11 December 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
DZDewei Zhou
Dewei Zhou
YLYou Li
You Li
FMFan Ma
Fan Ma
ZYZongxin Yang
Zongxin Yang
YYYi Yang
Yi Yang
We introduce the Multi-Instance Generation (MIG) task, which focuses on generating multiple instances within a single image, each accurately placed at predefined positions with attributes such as category, color, and shape, strictly following user specifications. MIG faces three main challenges: avoiding attribute leakage between instances, supporting diverse instance descriptions, and maintaining consistency in iterative generation. To address attribute leakage, we propose the Multi-Instance Generation Controller (MIGC). MIGC generates multiple instances through a divide-and-conquer strategy, breaking down multi-instance shading into single-instance tasks with singular attributes, later integrated. To provide more types of instance descriptions, we developed MIGC++. MIGC++ allows attribute control through text & images and position control through boxes & masks. Lastly, we introduced the Consistent-MIG algorithm to enhance the iterative MIG ability of MIGC and MIGC++. This algorithm ensures consistency in unmodified regions during the addition, deletion, or modification of instances, and preserves the identity of instances when their attributes are changed. We introduce the COCO-MIG and Multimodal-MIG benchmarks to evaluate these methods. Extensive experiments on these benchmarks, along with the COCO-Position benchmark and DrawBench, demonstrate that our methods substantially outperform existing techniques, maintaining precise control over aspects including position, attribute, and quantity. Project page: https://github.com/limuloo/MIGC .