Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

Federated learning (FL) has emerged as an efficient machine learning (ML) method with crucial privacy protection features. It is adapted for training models in Internet of Things (IoT)-related domains, including smart healthcare systems (SHSs), where the introduction of IoT devices and technologies can arise various security and privacy concerns. However, as FL cannot solely address all privacy challenges, privacy-enhancing technologies (PETs) and blockchain are often integrated to enhance privacy protection in FL frameworks within SHSs. The critical questions remain regarding how these technologies are integrated with FL and how they contribute to enhancing privacy protection in SHSs. This survey addresses these questions by investigating the recent advancements on the combination of FL with PETs and blockchain for privacy protection in smart healthcare. First, this survey emphasizes the critical integration of PETs into the FL context. Second, to address the challenge of integrating blockchain into FL, it examines three main technical dimensions such as blockchain-enabled model storage, blockchain-enabled aggregation, and blockchain-enabled gradient upload within FL frameworks. This survey further explores how these technologies collectively ensure the integrity and confidentiality of healthcare data, highlighting their significance in building a trustworthy SHS that safeguards sensitive patient information.

Generating radiology reports automatically reduces the workload of radiologists and helps the diagnoses of specific diseases. Many existing methods take this task as modality transfer process. However, since the key information related to disease accounts for a small proportion in both image and report, it is hard for the model to learn the latent relation between the radiology image and its report, thus failing to generate fluent and accurate radiology reports. To tackle this problem, we propose a memory-based cross-modal semantic alignment model (MCSAM) following an encoder-decoder paradigm. MCSAM includes a well initialized long-term clinical memory bank to learn disease-related representations as well as prior knowledge for different modalities to retrieve and use the retrieved memory to perform feature consolidation. To ensure the semantic consistency of the retrieved cross modal prior knowledge, a cross-modal semantic alignment module (SAM) is proposed. SAM is also able to generate semantic visual feature embeddings which can be added to the decoder and benefits report generation. More importantly, to memorize the state and additional information while generating reports with the decoder, we use learnable memory tokens which can be seen as prompts. Extensive experiments demonstrate the promising performance of our proposed method which generates state-of-the-art performance on the MIMIC-CXR dataset.

Vertical federated learning (VFL) provides a privacy-preserving method for machine learning, enabling collaborative training across multiple institutions with vertically distributed data. Existing VFL methods assume that participants passively gain local models of the same structure and communicate with active pary during each training batch. However, due to the heterogeneity of participating institutions, VFL with heterogeneous models for efficient communication is indispensable in real-life scenarios. To address this challenge, we propose a new VFL method called Practical Heterogeneous Vertical Federated Learning via Representation Learning (PraVFed) to support the training of parties with heterogeneous local models and reduce communication costs. Specifically, PraVFed employs weighted aggregation of local embedding values from the passive party to mitigate the influence of heterogeneous local model information on the global model. Furthermore, to safeguard the passive party’s local sample features, we utilize blinding factors to protect its local embedding values. To reduce communication costs, the passive party performs multiple rounds of local pre-model training while preserving label privacy. We conducted a comprehensive theoretical analysis and extensive experimentation to demonstrate that PraVFed reduces communication overhead under heterogeneous models and outperforms other approaches. For example, when the target accuracy is set at 60% under the CINIC10 dataset, the communication cost of PraVFed is reduced by 70.57% compared to the baseline method. Our code is available at https://github.com/wangshuo105/PraVFed_main.

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue task involves multiple rounds of dialogues which cover a broad range of visual content that could be related to any objects, relationships or high-level semantics. Thus one of the key challenges in Visual Dialogue task is to learn a more comprehensive and semantic-rich image representation that can adaptively attend to the visual content referred by variant questions. In this paper, we first propose a novel scheme to depict an image from both visual and semantic views. Specifically, the visual view aims to capture the appearance-level information in an image, including objects and their visual relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Furthermore, on top of such dual-view image representations, we propose a Dual Encoding Visual Dialogue (DualVD) module, which is able to adaptively select question-relevant information from the visual and semantic views in a hierarchical mode. To demonstrate the effectiveness of DualVD, we propose two novel visual dialogue models by applying it to the Late Fusion framework and Memory Network framework. The proposed models achieve state-of-theart results on three benchmark datasets. A critical advantage of the DualVD module lies in its interpretability. We can analyze which modality (visual or semantic) has more contribution in answering the current question by explicitly visualizing the gate values. It gives us insights in understanding of information selection mode in the Visual Dialogue task. The code is available at https://github.com/JXZe/Learning_DualVD.

The education sector has witnessed a significant shift towards digitising student records, with relevant data now stored in centralized data repositories. While traditional identity management solutions in education are functional, they often face various challenges, including data privacy concerns, limited portability, and reliability challenges. As the volume of student data continues to grow, inadequate data management practices have led to several problems. These include students losing control and empowerment over their educational information, increased vulnerability to potential data breaches and unauthorized access, a lack of transparency and accountability, data silos and inconsistencies, and administrative inefficiencies. To address these limitations, the implementation of a blockchain-assisted self-sovereign identity (Ba-SSI) concept in the education system presents a viable solution. Self-sovereign identity (SSI) represents a paradigm shift from traditional centralized identity systems, allowing individuals to maintain full control of their identity data without relying on centralized authorities. By leveraging the decentralized nature, SSI frameworks can ensure security, interoperability, and scalability, thereby improving user-centric identity management. This survey paper explores the potential of Ba-SSI within the context of education. It thoroughly reviews the current state of digital identity management in education, highlighting the limitations of conventional systems and the emerging role of blockchain technology in addressing these challenges. The paper discusses the fundamental principles of blockchain technology and how it can be utilized to enhance security, interoperability, and scalability in identity management. Additionally, it examines the insights and benefits of this approach for the education system. Finally, the paper concludes by addressing the issues, challenges, benefits, and future research directions in this domain, underscoring the potential of Ba-SSI solutions to revolutionize the management and empowerment of student data within the education sector.