Sitong Su

Sitong Su

Auto-generated profile, based on publicly available information

Sitong Su

Center for Future Media and School of Computer Science and Engineering, University of Electronic Science and Technology of China, China

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 July 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
SSSitong Su
Sitong Su
JZJunchen Zhu
Junchen Zhu
LGLianli Gao
Lianli Gao
JSJingkuan Song
Jingkuan Song
Multimodal Conditional Image Synthesis(MCIS) aims to generate images according to different modalities input and their combination, which allows users to describe their requirements in complementary ways, e.g. segmentation for shapes and text for attributes. Despite satisfying results in MCIS, a non-trivial issue is neglected. Some modalities are fully optimized and dominate the generation, while other modalities are sub-optimized and fail to contribute their complementary information. We coin this phenomenon as Modality Bias. Our analysis reveals that generative models own greedy nature . Specifically, the modality that shares less semantic gap with the synthesized modality will be greedily incorporated and thus takes a larger proportion in synthesis. The main idea of previous works in Modality Bias is to punish the greedy nature, which hurts the performance of dominant modalities and impedes their contribution to multimodal synthesis. Instead, we propose to utilize the greedy nature by setting dominant modalities as guidance for sub-optimized modalities through coordinated feature space, named Coordinated Knowledge Mining. Afterwards, improved uni-modalities are aggregated by fusing coordinated features to further boost the performance of multimodal image synthesis, called Coordinated Knowledge Fusion. Extensive experiments prove that our method not only increases uni-modal performance by a large margin, but also promotes multimodal image synthesis by fully utilizing complementary information from different modalities.

4 November 2022 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
LGLianli Gao
Lianli Gao
QZQike Zhao
Qike Zhao
JZJunchen Zhu
Junchen Zhu
SSSitong Su
Sitong Su
LCLechao Cheng
Lechao Cheng
LZLei Zhao
Lei Zhao
Manipulating visual attributes of an image through a natural language description, known as text-to-image attributes manipulation (T2AM), is a challenging task. However, existing approaches tend to search the whole image to manipulate the target instance indicated by a description, thus they often fail to locate and manipulate the accurate text-relevant regions, and even disturb the text-irrelevant contents, e.g. texture and background. Meanwhile, the model efficiency needs to be improved. To tackle the above issues, we introduce a novel yet simple GAN-based approach, namely Structuring Image for Manipulating (SIMGAN), to narrow down the optimization areas from external to internal. It consists of two major components: 1) External Structuring (ExST), a pretrained segmentation network, for recognizing and separating the target instances and background from an image; and 2) Internal Structuring (InST) for seeking out and editing the text-relevant attributes of the target instances based on the given description and masked hierarchical image representations from ExST. Specifically, the InST structures target instances from outline to detail by firstly drawing the sketch and colors underpainting of instances with an Outline-Oriented Structuring (OuST), and then enhancing the text-relevant attributes and elaborating on details with a Detail-Oriented Structuring (DeST). Extensive experiments on benchmark datasets demonstrate that our framework significantly outperforms state-of-the-art both quantitatively and qualitatively. Compared with the state-of-the-art method ManiGAN, our approach reduces the training time by 88%, while the inferring time is three times faster. In addition, our approach is easily extended to solve the instance-level image-to-image translation problem, and the results exhibit the versatility and effectiveness of our approach. We release our code in https://github.com/qikizh/SIMGAN .

18 December 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
YLYu Liu
Yu Liu
SSSitong Su
Sitong Su
JZJunchen Zhu
Junchen Zhu
FZFeng Zheng
Feng Zheng
LGLianli Gao
Lianli Gao
JSJingkuan Song
Jingkuan Song
Replacing objects in images is a practical functionality of Photoshop, e.g., clothes changing. This task is defined as Unsupervised Deformable-Instances Image-to-Image Translation (UDIT), which maps multiple foreground instances of a source domain to a target domain, involving significant changes in shape. Although previous works incorporate instance masks of source domain for instance shape indication, their translation still fails in shape because of inadequate utilization of shape information in masks. To mitigate this issue, we introduce an effective two-stage pipeline for UDIT called Mask-Guided Deformable-instances GAN++ (MGD-GAN++), which generates target masks in the first stage named Mask Morph and utilizes the masks to guide the synthesis of corresponding instances in the second stage named Mask-Guided Image Generation. To further provide sufficient supervision with existing unpaired datasets, an overall set of training schemes is proposed for the two stages of MGD-GAN++, coined as Aligned Supervision and Inpainting Supervision, respectively. Extensive experiments on four datasets demonstrate the significant advantages of our MGD-GAN++ over existing methods both quantitatively and qualitatively. Furthermore, our training time consumption is hugely reduced compared to the state-of-the-art.

12 March 2026 in IEEE Transactions on Image Processing
Verified
Journal Article
Closed access
XCXiao Cai
Xiao Cai
PZPengpeng Zeng
Pengpeng Zeng
LGLianli Gao
Lianli Gao
SSSitong Su
Sitong Su
HSHeng Tao Shen
Heng Tao Shen
JSJingkuan Song
Jingkuan Song
General Text-to-3D (GT23D) generation is crucial for creating diverse 3D content across objects and scenes, yet it faces two key challenges: 1) ensuring semantic consistency between input text and generated 3D models, and 2) maintaining multi-view consistency across different perspectives within 3D. Existing approaches typically address only one of these challenges, often leading to suboptimal results in semantic fidelity and structural coherence. To overcome these limitations, we propose SeMv-3D, a novel framework that jointly enhances semantic alignment and multi-view consistency in GT23D generation. At its core, we introduce Triplane Prior Learning (TPL), which effectively learns triplane priors by capturing spatial correspondences across three orthogonal planes using a dedicated Orthogonal Attention mechanism, thereby ensuring geometric consistency across viewpoints. Additionally, we present Prior-based Semantic Aligning in Triplanes (SAT), which enables consistent any-view synthesis by leveraging attention-based feature alignment to reinforce the correspondence between textual semantics and triplane representations. Extensive experiments demonstrate that our method sets a new state-of-the-art in multi-view consistency, while maintaining competitive performance in semantic consistency compared to methods focused solely on semantic alignment. These results emphasize the remarkable ability of our approach to effectively balance and excel in both dimensions, establishing a new benchmark in the field.