Magnetic resonance imaging (MRI) can provide multi-modality MR images by setting task-specific scan parameters, and has been widely used in various disease diagnosis and planned treatments. However, in practical clinical applications, it is often difficult to obtain multi-modality MR images simultaneously due to patient discomfort, and scanning costs, etc. Therefore, how to effectively utilize the existing modality images to synthesize missing modality image has become a hot research topic. In this paper, we propose a novel confidence-guided aggregation and cross-modality refinement network (CACR-Net) for multi-modality MR image synthesis, which effectively utilizes complementary and correlative information of multiple modalities to synthesize high-quality target-modality images. Specifically, to effectively utilize the complementary modality-specific characteristics, a confidence-guided aggregation module is proposed to adaptively aggregate the multiple target-modality images generated from multiple source-modality images by using the corresponding confidence maps. Based on the aggregated target-modality image, a cross-modality refinement module is presented to further refine the target-modality image by mining correlative information among the multiple source-modality images and aggregated target-modality image. By training the proposed CACR-Net in an end-to-end manner, high-quality and sharp target-modality MR images are effectively synthesized. Experimental results on the widely used benchmark demonstrate that the proposed method outperforms state-of-the-art methods.
One of the challenges in emotion recognition is finding an effective way to represent spatial-temporal features from EEG. To fully utilize the features on multiple dimensions of EEG signals, we propose a parallel sequence-channel projection convolutional neural network, including temporal stream sub-network, spatial stream sub-network, and fusion classification block. Temporal stream extracts temporal continuity via sequence-projection layer while spatial stream captures spatial correlation via channel-projection layer. Both sequence-projection and channel-projection adopt length-synchronized convolutional kernel to decode whole time and space information. The size of length-synchronized convolutional kernel is equal to the length of transmitted EEG sequence. The fusion classification block combines the extracted temporal and spatial features into a joint spatial-temporal feature vector for emotion prediction. In addition, we present a baseline noise filtering module to amplify input signals and a random channels exchange strategy to enrich the baseline-removed emotional signals. Experimental evaluation on DEAP dataset reveals that the proposed method achieves state-of-the-art classification performance for the binary classification task. The recognition accuracies reach to 96.16% and 95.89% for valence and arousal. The proposed method can improve 3% to 6% than other latest advanced works.
12 May 2023 in IEEE Transactions on Emerging Topics in Computational Intelligence
With the popular application of convolutional neural networks in computational intelligence, research on deep learning-based view synthesis has been a hot topic. Although promising performance has been achieved by the existing learning-based view synthesis methods, how to obtain a clearer target view in the single-view synthesis task is still a challenging problem. In this paper, we propose a novel deep gradual-conversion and cycle network (DGCC-Net) for single-view synthesis by jointly considering the gradual and cycle synthesis between source and target views. Specifically, a gradual conversion mechanism is designed to synthesize a clearer target view in a gradual manner, which learns the progressive rotation trend from the source to the target view by introducing the intermediate transformation. Based on the synthesized target view, a cycle synthesis mechanism is designed to further promote the learning of single-view synthesis network by mapping the synthesized target back to the source view. By utilizing the proposed gradual conversion and cycle synthesis mechanisms, the whole network achieves a cycle view synthesis mapping between source and target views to obtain a better target view. Experiments on widely used datasets indicate the proposed DGCC-Net exceeds state-of-the-art methods.
19 June 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Existing unsupervised Multi-View Stereo (MVS) methods generally construct supervision on the basis of the photometric consistency loss, which suffers from unreliable supervision and limited scalability. In this paper, a novel unsupervised MVS framework with Self-constructed Stereo Correspondences, termed SSC-MVS, is proposed to provide reliable supervision for the network and improve scalability of unsupervised MVS. Specifically, a pseudo depth-based learning strategy is first presented to supervise the MVS network with a pseudo depth, which is used to characterize the accurate stereo correspondences. Additionally, a consistency-based training mechanism is designed, where the depth consistency between two differently-augmented inputs is constrained to further improve the robustness of the network in real MVS scenes. Experimental results on widely-used MVS datasets demonstrate that the proposed SSC-MVS obtains the state-of-the-art performance among the unsupervised methods and has the potential to outperform the fully-supervised methods. The code is available at https://github.com/jzhu98/ssc-mvs.
13 June 2024 in ACM Transactions on Multimedia Computing, Communications, and Applications
Monocular depth estimation aims to infer a depth map from a single image. Although supervised learning-based methods have achieved remarkable performance, they generally rely on a large amount of labor-intensively annotated data. Self-supervised methods, on the other hand, do not require any annotation of ground-truth depth and have recently attracted increasing attention. In this work, we propose a self-supervised monocular depth estimation network via binocular geometric correlation learning. Specifically, considering the inter-view geometric correlation, a binocular cue prediction module is presented to generate the auxiliary vision cue for the self-supervised learning of monocular depth estimation. Then, to deal with the occlusion in depth estimation, an occlusion interference attenuated constraint is developed to guide the supervision of the network by inferring the occlusion region and producing paired occlusion masks. Experimental results on two popular benchmark datasets have demonstrated that the proposed network obtains competitive results compared to state-of-the-art self-supervised methods and achieves comparable results to some popular supervised methods.
12 July 2023 in IEEE Transactions on Circuits and Systems for Video Technology
View synthesis aims to learn a view transformation and synthesize the target views from a single or multiple source views. Although previous view synthesis methods have obtained promising performance, they heavily rely on the supervision of the target view. In this paper, we propose an unsupervised single-view synthesis network (USVS-Net) to learn the view transformation without the supervision of the target view. Specifically, with the usage of only a single source view, a style-guidance view synthesis model is proposed to learn an intrinsic representation, which intends to describe the object from a reference pose. With the intrinsic representation, the view transformation is learned to boost the learning of the unsupervised single-view synthesis. Then, taking the style-guidance view synthesis model as the teacher, a prior-distillation view synthesis model is further presented as the student to learn a more direct view transformation. By utilizing the proposed method, high-quality target views are synthesized in a time-efficient manner. Experiments on both synthetic and real-scene datasets show that despite the lack of supervision of the target view, the proposed method achieves promising results compared with the existing view synthesis methods.