Multimodal Artificial Intelligence (Multimodal AI), in general, involves various types of data (e.g., images, texts, or data collected from different sensors), feature engineering (e.g., extraction, combination/fusion), and decision-making (e.g., majority vote). As architectures become more and more sophisticated, multimodal neural networks can integrate feature extraction, feature fusion, and decision-making processes into one single model. The boundaries between those processes are increasingly blurred. The conventional multimodal data fusion taxonomy (e.g., early/late fusion), based on which the fusion occurs in, is no longer suitable for the modern deep learning era. Therefore, based on the main-stream techniques used, we propose a new fine-grained taxonomy grouping the state-of-the-art (SOTA) models into five classes: Encoder-Decoder methods, Attention Mechanism methods, Graph Neural Network methods, Generative Neural Network methods, and other Constraint-based methods. Most existing surveys on multimodal data fusion are only focused on one specific task with a combination of two specific modalities. Unlike those, this survey covers a broader combination of modalities, including Vision + Language (e.g., videos, texts), Vision + Sensors (e.g., images, LiDAR), and so on, and their corresponding tasks (e.g., video captioning, object detection). Moreover, a comparison among these methods is provided, as well as challenges and future directions in this area.

In recent decades, millions of people are killed by natural disasters such as wildfire, landslide, tsunami, and volcanic eruption. The efficiency of post-disaster emergency responses and humanitarian assistance has become crucial in minimizing the expected casualties. This paper focuses on the task of building damage level evaluation, which is a key step for maximizing the deployment efficiency of post-event rescue activities. In this paper, we implement a Mask R-CNN based building damage evaluation model with a practical two-stage training strategy. The motivation of Stage-l is to train a ResNet 101 backbone in Mask R-CNN as a Building Feature Extractor. In Stage-2, we further build on top the model trained in Stage-l a deep learning architecture that performs more sophisticated tasks and is able to classify buildings with different damage levels from satellite images. In particular, in order to take advantage of pre-disaster satellite images, we extract the ResNet 101 backbone from the Mask R-CNN trained on pre-disaster images in Stage-l and utilize it to build a Siamese based semantic segmentation model for classifying the building damage level at the pixel level. The pre- and post-disaster satellite images are simultaneously fed into the proposed Siamese based model during the training and inference process. The output of these two models own the same size as input satellite images. Buildings with different damage levels, i.e., ‘no damage’, ‘minor damage’, ‘major damage’, and ‘destroyed’, are represented as segments of different damage classes in the output. Comparative experiments are conducted on the xBD satellite imagery dataset and compared with multiple state-of-the-art methods. The experimental results indicate that the proposed Siamese based method is capable to improve the damage evaluation accuracy by 16 times and 80%, compared with a baseline model implemented by xBD team and the Mask-RCNN framework, respectively.

Introduction: Orthopaedic applicants have increased the average number of publications on their residency application to compete with the growing competitiveness of the field. The purpose of this study was to assess whether research productivity before orthopaedic residency and caliber of one's institution is correlated with academic productivity during residency. Methods: Scopus was used to extract publication metrics. Quantity and quality (how often the publications were cited) were analyzed at 2 different time periods: before and during residency. All subjects in the study had graduated an ACGME-accredited orthopaedic surgery residency in 2021. Military residents, international medical graduates, and residents not listed on their department's website were excluded. Residents were categorized as both high (≥2 publications) or low (<2 publications) publishers according to their pre-residency publications. They were also categorized based on their program's Jones et al. research productivity ranking. Results: For the 758 residents, the median number of publications was 0 (Interquartile Range [IQR]: 0-2) and 3 (IQR: 1-6) before and during residency, respectively. High publishing medical students had more publications during training than low publishers (6 [IQR: 3-14] and 2 [IQR: 1-4], p < 0.001). Residents at higher ranked programs also had more publications (4 [IQR: 2-9] and 2 [IQR: 0-4], p < 0.001). High publishing students now training at lower ranked institutions had more publications during residency than low publishers who trained at more productive institutions (4 [IQR: 1-9] and 3 [IQR: 1-6], p < 0.001). Conclusion: Having 2 or more publications before residency is correlated with an increased number of publications during residency. While attending a higher academically productive program is associated with increased resident publications, a high publishing medical student would be expected to have more publications during residency than a low publishing student, regardless of program rank. Notably, most matched applicants continue to have zero publications before matriculation.

In response to the urgent need for rapid and precise post-disaster damage evaluation, this study introduces the Visual Prompt Damage Evaluation (ViPDE) framework, a novel contrastive learning-based approach that leverages the embedded knowledge within the Segment Anything Model (SAM) and pairs of remote sensing images to enhance building damage assessment. In this framework, we propose a learnable cascaded Visual Prompt Generator (VPG) that provides semantic visual prompts, guiding SAM to effectively analyze pre- and post-disaster image pairs and construct a nuanced representation of the affected areas at different stages. By keeping the foundation model’s parameters frozen, ViPDE significantly enhances training efficiency compared with traditional full-model fine-tuning methods. This parameter-efficient approach reduces computational costs and accelerates deployment in emergency scenarios. Moreover, our model demonstrates robustness across diverse disaster types and geographic locations. Beyond mere binary assessments, our model distinguishes damage levels with a finer granularity, categorizing them on a scale from 1 (no damage) to 4 (destroyed). Extensive experiments validate the effectiveness of ViPDE, showcasing its superior performance over existing methods. Comparative evaluations demonstrate that ViPDE achieves an F1 score of 0.7014. This foundation model-based approach sets a new benchmark in disaster management. It also pioneers a new practical architectural paradigm for foundation model-based contrastive learning focused on specific objects of interest.

Currently deployed election systems that scan and process hand-marked ballots are not sophisticated enough to handle marks insufficiently filled in (e.g., partially filled-in), improper marks (e.g., using check marks or crosses instead of filling in bubbles), or marks outside of bubbles, other than setting a threshold to detect whether the pixels inside bubbles are dark and dense enough to be counted as a vote. The current works along this line are still largely limited by their degree of automation and require substantial manpower for annotation and adjudication. In this study, we propose a highly automated deep learning (DL) mark segmentation model-based ballot tabulation assistant able to accurately identify legitimate ballot marks. For comparison purposes, a highly customized traditional computer vision (T-CV) mark segmentation-based method has also been developed to compare with the DL-based tabulator, with a detailed discussion included. Our experiments conducted on two real election datasets achieved the highest accuracy of 99.984% on ballot tabulation. In order to further enhance our DL model’s capability of detecting the marks that are underrepresented in training datasets, e.g., insufficiently or improperly filled marks, we propose a Siamese network architecture that enables our DL model to exploit the contrasting features between a hand-marked ballot image and its corresponding blank template image to detect marks. Without the need for extra data collection, by incorporating this novel network architecture, our DL model-based tabulation method not only achieved a higher accuracy score but also substantially reduced the overall false negative rate.

The emergency of anonymous encrypted digital currency based on blockchain brings the rapid growth of financial crimes simultaneously. However, under the condition of Know Your Customer rules, the traditional rule-based filtering and supervised pattern recognition methods are mainly built, which does not apply to the scenario of anonymous encrypted digital currency. In this paper, we attempt to tackle this problem by constructing user graph from transactions and dividing the whole user graph into tightly connected communities and clustering similar communities into groups. Experimental results on bitcoin transaction datasets show that the proposed approach has higher than 92% precision and higher than 73% recall for identifying gambling and mining pool communities.