Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions and generate results with undesired properties or even harmful content. Inspired by the success and popularity of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences. This work mainly reviews alignment of diffusion models, covering advancements in fundamentals of alignment, alignment techniques of diffusion models, preference benchmarks, and evaluation for diffusion models. Moreover, we discuss key perspectives on current challenges and promising future directions on solving the remaining challenges in alignment of diffusion models. To the best of our knowledge, our work is the first comprehensive review paper for researchers and engineers to comprehend, practice, and research alignment of diffusion models.
1 November 2023 in IEEE Transactions on Knowledge and Data Engineering
Graphs in real-world applications usually evolve constantly presenting dynamic behaviors such as social networks and transportation networks. Hence, dynamic graph embedding has gained much attention recently. In dynamic graphs, both the topology and node attributes could change over time, which pose great challenges for developing effective embedding models. Typically, the evolution process of a dynamic graph can be recorded as a series of snapshots. We observe that the evolution process inherently provides both prior information (previous snapshots) and validation information (the next snapshot). The prior information can be used to fit the evolution process, while the validation information can be used to improve the generalization ability of a graph embedding model. However, existing dynamic graph embedding models only utilize the prior information, but overlook the validation information. To tackle this issue, this paper proposes a novel dynamic graph embedding method via Model-Agnostic Meta-Learning, which utilizes both kinds of information to obtain better graph representation. The extensive experiments on eight real-world datasets demonstrate the superiority of our proposed method over state-of-the-art methods on various graph analysis tasks.
Personalized federated learning (PFL) is a variant of federation learning, which improves the model performance of each participant through collaborative training while providing a customized local model that meets their unique needs. Balancing global (general) knowledge and local (personalized) knowledge is a core challenge in PFL. Existing PFL methods neglect knowledge forgetting during aggregation and updating, impacting the fusion of both types of knowledge. We empirically confirm the existence of knowledge forgetting, which leads to performance degradation during FL. This observation motivates us to rethink PFL from a knowledge perspective. Therefore, we propose Personalized Federated Learning Framework with Adaptive Model Fusion (pFedAMF). We consider the global model as global knowledge and history state models as local knowledge, and preserve them before forgetting occurs. After local training, we use an adaptive knowledge matrix to fuse knowledge from local, global, and history state models. This fused knowledge is then distilled into an individual model, thus transferring the fused knowledge into global models. Extensive experiments show that pFedAMF consistently outperforms FedAvg, achieving up to 5.22% average accuracy boost, and reducing computation cost by 82% and communication cost by 87%.
24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Defocus blur, due to spatially-varying sizes and shapes, is hard to remove. Existing methods either are unable to effectively handle irregular defocus blur or fail to generalize well on other datasets. In this work, we propose a divide-and-conquer approach to tackling this issue, which gives rise to a novel end-to-end deep learning method, called prior-and-prediction inverse kernel transformer (P2IKT), for single image defocus deblurring. Since most defocus blur can be approximated as Gaussian blur or its variants, we construct an inverse Gaussian kernel module in our method to enhance its generalization ability. At the same time, an inverse kernel prediction module is introduced in order to flexibly address the irregular blur that cannot be approximated by Gaussian blur. We further design a scale recurrent transformer, which estimates mixing coefficients for adaptively combining the results from the two modules and runs the scale recurrent ``coarse-to-fine" procedure for progressive defocus deblurring. Extensive experimental results demonstrate that our P2IKT outperforms previous methods in terms of PSNR on multiple defocus deblurring datasets.
28 October 2024 in IEEE Transactions on Knowledge and Data Engineering
Existing geometric knowledge graph embedding methods employ various relational transformations, such as translation, rotation, and projection, to model different relation patterns, which aims to enhance the expressiveness of models. In contrast to current approaches that treat the expressiveness of the model as a binary issue, we aim to delve deeper into analyzing the level of difficulty in which geometric knowledge graph embedding models can represent relation patterns. In this paper, we provide a theoretical analysis framework that measures the expressiveness of the model in relation patterns by quantifying the size of the solution space of linear equation systems. Additionally, we propose a mechanism for imposing relational constraints on geometric knowledge graph embedding models by setting “traps” near relational optimal solutions, which enables the model to better converge to the optimal solution. Empirically, we analyze and compare several typical knowledge graph embedding models with different geometric algebras, revealing that some models have insufficient solution space due to their design, which leads to performance weaknesses. We also demonstrate that the proposed relational constraint operations can improve the performance of certain relation patterns. The experimental results on public benchmarks and relation pattern specified dataset are consistent with our theoretical analysis.
17 July 2025 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
As a result of global warming, the intensity and frequency of extreme precipitation events have increased, posing significant threats to human life and property. Currently, precipitation nowcasting methods based on long short-term memory (LSTM) or Vision Transformers (ViT) are becoming mainstream. While they accurately predict ordinary precipitation events, these methods often fall short in nowcasting extreme precipitation events. This discrepancy can be attributed to the substantially higher intensity of extreme precipitation relative to ordinary precipitation, which presents greater challenges for accurate nowcasting. However, the existing methods tend to inadequately weigh precipitation intensity features by only implicitly learning and modeling these features within the spatial distribution. To address this issue, we design an Intensity Trend Module (ITM) to explicitly learn and model the intensity features of extreme precipitation events. ITM employs intensity trend factors to capture the intensity features of extreme precipitation, thereby mitigating intensity misestimation. Moreover, the current architectures of LSTM and ViT often compromise the spatial structure of precipitation, thereby contributing to the inadequate performance in the nowcasting of extreme precipitation events. Hence, we also design a Spatio-Temporal Coherence Module (STCM), which exploits voxel flow to capture spatio-temporal coherence features. Subsequently, by processing multitemporal state features and integrating coherence features, STCM can preserve the spatio-temporal structure of extreme precipitation events. Finally, building upon ITM and STCM, we propose a framework of Joint Intensity and Spatio-Temporal Representation Learning for Extreme Precipitation Nowcasting. Experimental results indicate that our method is capable of nowcasting up to 4 hours. Across three datasets, our method surpasses state-of-the-art methods by achieving an average increase of 10.28% in the Critical Success Index at the highest threshold, and an average improvement of 81.7% in Structural Similarity.