5 February 2024 in IEEE Transactions on Neural Networks and Learning Systems
We propose two novel transferability metrics fast optimal transport-based conditional entropy (F-OTCE) and joint correspondence OTCE (JC-OTCE) to evaluate how much the source model (task) can benefit the learning of the target task and to learn more generalizable representations for cross-domain cross-task transfer learning. Unlike the original OTCE metric that requires evaluating the empirical transferability on auxiliary tasks, our metrics are auxiliary-free such that they can be computed much more efficiently. Specifically, F-OTCE estimates transferability by first solving an optimal transport (OT) problem between source and target distributions and then uses the optimal coupling to compute the negative conditional entropy (NCE) between the source and target labels. It can also serve as an objective function to enhance downstream transfer learning tasks including model finetuning and domain generalization (DG). Meanwhile, JC-OTCE improves the transferability accuracy of F-OTCE by including label distances in the OT problem, though it incurs additional computation costs. Extensive experiments demonstrate that F-OTCE and JC-OTCE outperform state-of-the-art auxiliary-free metrics by $21.1\%$ and $25.8\%$ , respectively, in correlation coefficient with the ground-truth transfer accuracy. By eliminating the training cost of auxiliary tasks, the two metrics reduce the total computation time of the previous method from 43 min to 9.32 and 10.78 s, respectively, for a pair of tasks. When applied in the model finetuning and DG tasks, F-OTCE shows significant improvements in the transfer accuracy in few-shot classification experiments, with up to $4.41\%$ and $2.34\%$ accuracy gains, respectively.
1 September 2026 in Open Conference Proceedings
Scientific figures encode critical experimental findings that are often not fully captured in textual descriptions. Despite recent advances in MLLMs, their performance on domain-specific scientific figures remains limited due to insufficient grounding in relevant context. In this work, we focus on in Task 3 (summarization) and Task 4 (visual question answering) of the Sci-ImageMiner 2026 challenge. We propose a structured context augmentation approach that improves model performance by providing relevant and well-organized contextual inputs. Our method extracts figure-centric and cross-referenced textual evidence from scientific papers and integrates them into structured prompts for model training. Experimental results show that our approach consistently improves performance across both tasks. We achieve first place in both Task 3 and Task 4, demonstrating the effectiveness of structured contextual inputs for scientific figure reasoning. Code is available at https://github.com/yuanjun0416/SciImageMiner34