Visual encoding models often use deep neural networks to describe the brain’s visual cortex response to external stimuli. Inspired by biological findings, researchers found that large receptive fields built with large convolutional kernels improve convolutional encoding model performance. Inspired by scaling laws in recent years, this article investigates the performance of large convolutional kernel encoding models on larger parameter scales. This paper proposes a large-scale parameters framework with a sizeable convolutional kernel for encoding visual functional magnetic resonance imaging activity information. The proposed framework consists of three parts: First, the stimulus image feature extraction module is constructed using a large-kernel convolutional network while increasing channel numbers to expand the parameter size of the framework. Second, enlarging the input data during the training stage through the multi-subject fusion module to accommodate the increase in parameters. Third, the voxel mapping module maps from stimulus image features to functional magnetic resonance imaging signals. Compared to sizeable convolutional kernel visual encoding networks with base parameter scale, our visual encoding framework improves by approximately 7% on the Natural Scenes Dataset, the dedicated dataset for the Algonauts 2023 Challenge. We further analyze that our encoding framework made a trade-off between encoding performance and trainability. This paper confirms that expanding parameters in visual coding can bring performance improvements.
Research on visual encoding models for functional magnetic resonance imaging derived from deep neural networks, especially CNN (e.g., VGG16), has been developed. However, CNNs typically use smaller kernel sizes (e.g., 3 × 3) for feature extraction in visual encoding models. Although the receptive field size of CNN can be enlarged by increasing the network depth or subsampling, it is limited by the small size of the convolution kernel, leading to an insufficient receptive field size. In biological research, the size of the neuronal population receptive field of high-level visual encoding regions is usually three to four times that of low-level visual encoding regions. Thus, CNNs with a larger receptive field size align with the biological findings. The RepLKNet model directly expands the convolution kernel size to obtain a larger-scale receptive field. Therefore, this paper proposes a mixed model to replace CNN for feature extraction in visual encoding models. The proposed model mixes RepLKNet and VGG so that the mixed model has a receptive field of different sizes to extract more feature information from the image. The experimental results indicate that the mixed model achieves better encoding performance in multiple regions of the visual cortex than the traditional convolutional model. Also, a larger-scale receptive field should be considered in building visual encoding models so that the convolution network can play a more significant role in visual representations.
Transferability of adversarial samples under different convolutional neural network (CNN) models is one of the metrics indicators to assess the efficiency of adversarial examples and an important research direction in defense of that. Transferability isolate models employ a particular alternative model to avoid black-box attacks. Meanwhile, recent research has revealed that adversarial transferability across sub-models may be utilized to express the diversity requirements of sub-models under ensemble robustness abstractly. Due to the lack of mathematical description for this adversarial transferability, it was utilized to be abstractly described as the diversity of different hypotheses. This paper employs the Jacobians matrix’s singular value decomposition (SVD) to provide a more accurate mathematical description of the transferability of adversarial samples between models and proposes a corresponding evaluation metric. Based on this metric, a new regularization constraint is introduced into the ensemble training, and the adversarial transferability between the sub-models is isolated optimally without the prior information of the adversarial samples. Based on the proposed metric accurately defining the transferability, further ensemble robustness experiments under small-scale dataset disentangle the correlation between transferability and robustness, indicating that the transferability isolation can only achieve robustness under an alternative transfer-based attack with partial sub-models of the ensemble.
Biological research has revealed that the verbal semantic information in the brain cortex, as an additional source, participates in nonverbal semantic tasks, such as visual encoding. However, previous visual encoding models did not incorporate verbal semantic information, contradicting this biological finding. This paper proposes a multimodal visual information encoding network model based on stimulus images and associated textual information in response to this issue. Our visual information encoding network model takes stimulus images as input and leverages textual information generated by a text-image generation model as verbal semantic information. This approach injects new information into the visual encoding model. Subsequently, a Transformer network aligns image and text feature information, creating a multimodal feature space. A convolutional network then maps from this multimodal feature space to voxel space, constructing the multimodal visual information encoding network model. Experimental results demonstrate that the proposed multimodal visual information encoding network model outperforms previous models under the exact training cost. In voxel prediction of the left hemisphere of subject 1's brain, the performance improves by approximately 15.87%, while in the right hemisphere, the performance improves by about 4.6%. The multimodal visual encoding network model exhibits superior encoding performance. Additionally, ablation experiments indicate that our proposed model better simulates the brain's visual information processing.
Recently, there has been a surge in the popularity of pre trained large language models (LLMs) (such as GPT-4), sweeping across the entire Natural Language Processing (NLP) and Computer Vision (CV) communities. These LLMs have demonstrated advanced multi-modal understanding capabilities and showcased strong performance across various benchmarks. The LLM has started to embody traits of artificial general intelligence, which holds vital guidance for enhancing brain-like characteristics within visual encoding models. Hence, This paper proposes a new multi-modal training paradigm, aligning with LLM, for encoding fMRI activity in visual cortex. Based on this paradigm, we trained an encoding model in fMRI data named the LLM-Visual Encoding Model (LLM-VEM). Specifically, we utilize LLM (miniGPT4) to generate descriptive text for all stimulus images, forming a high-quality textual description set. Moreover, we use the pre-trained text encoder (CLIP) to process these detailed descriptions, obtaining the text embedding features. Next, we use the contrast loss function to minimize the distance between the image embedding features and the text embedding features to complete the alignment operation of the stimulus image and text information. With the assistance of the pre-trained LLM, this alignment process facilitates better learning of the visual encoding model, resulting in higher precision. The final experimental results indicate that our training paradigm has significantly aided in enhancing the performance of the visual encoding model.