Multimodal large language models (MLLMs) have experienced significant advancements recently, but still struggle to recognize and interpret intricate details in high-resolution (HR) images effectively. While state-of-the-art (SOTA) MLLMs claim to process images at 4K resolution, existing MLLM benchmarks only support up to 2K, leaving the capabilities of SOTA models on true HR images largely untested. Furthermore, existing methods for enhancing HR image perception in MLLMs rely on computationally expensive visual instruction tuning. To address these limitations, we introduce HR-Bench, the first deliberately designed benchmark to rigorously evaluate MLLM performance on 4K & 8K images. Through extensive experiments, we demonstrate that while downsampling HR images leads to vision information loss, leveraging complementary modalities, e.g., text, can effectively compensate for this loss. Building upon this insight, we propose Divide, Conquer and Combine, a novel training-free framework for enhancing MLLM perception of HR images. Our method follows a three-staged approach: 1) Divide: recursively partitioning the HR image into patches and merging similar patches to minimize computational overhead, 2) Conquer: leveraging the MLLM to generate accurate textual descriptions for each image patch, and 3) Combine: utilizing the generated text descriptions to enhance the MLLM's understanding of the overall HR image. Extensive experiments show that: 1) the SOTA MLLM achieves 63% accuracy, which is markedly lower than the 87% accuracy achieved by humans on HR-Bench; 2) our method brings consistent and significant improvements (a relative increase of +6% on HR-Bench and +8% on general multimodal benchmarks).
4 October 2018 in Physica A: Statistical Mechanics and its Applications
The rising popularity of tabular data in data science applications has led to a surge of interest in utilizing deep neural networks (DNNs) to address tabular problems. Existing deep neural network methods are not effective in handling two fundamental challenges that are inherent in tabular data: permutation invariance (where the labels remain unchanged regardless of element order) and local dependency (where predictive labels are solely determined by local features). Furthermore, given the inherent heterogeneity among elements in tabular data, effectively capturing heterogeneous feature interactions remains unresolved. In this paper, we propose a novel Multiplex Cross-Feature Interaction Network (MPCFIN) by explicitly and systematically modeling feature relations with interactive graph neural networks. Specifically, MPCFIN first learns the most relevant features associated with individual features, and merges them to form cross-feature embedding. Subsequently, we design a multiplex graph neural network to learn enhanced representation for each sample. Comprehensive experiments on seven datasets demonstrate that MPCFIN exhibits superior performance over deep neural network methods in modeling the tabular data, showcasing consistent interpretability in its cross-feature embedding module for medical diagnosis applications.
2 October 2018 in Concurrency and Computation: Practice and Experience