The explosive growth of videos on streaming media platforms has underscored the urgent need for effective video quality assessment (VQA) algorithms to monitor and perceptually optimize the quality of streaming videos. However, VQA remains an extremely challenging task due to the diverse video content and the complex spatial and temporal distortions, thus necessitating more advanced methods to address these issues. Nowadays, large multimodal models (LMMs), such as GPT-4V, have exhibited strong capabilities for various visual understanding tasks, motivating us to leverage the powerful multimodal representation ability of LMMs to solve the VQA task. Therefore, we propose an Large Multi-Modal based Video Quality Assessment (LMM-VQA) model, which introduces a novel spatiotemporal visual modeling strategy for quality-aware feature extraction. Specifically, we reformulate the quality regression problem into a question and answering (Q&A) task and construct Q&A prompts for VQA instruction tuning. Then, we design a spatiotemporal vision encoder to extract spatial and temporal features to represent the quality characteristics of videos, which are subsequently mapped into the language space by the spatiotemporal projector for modality alignment. Finally, the aligned visual tokens and the quality-inquired text tokens are aggregated as inputs for the large language model (LLM) to generate the quality score as well as the quality level. Extensive experiments demonstrate that LMM-VQA achieves state-of-the-art performance across five VQA benchmarks, exhibiting an average improvement of 5% in generalization ability over existing methods. Furthermore, due to the advanced design of the spatiotemporal encoder and projector, LMM-VQA also performs exceptionally well on general video understanding tasks, further validating its effectiveness. Our code will be released at https://github.com/Sueqk/LMM-VQA.
Near-infrared spectroscopy has been widely applied in various fields such as food analysis and agricultural testing. However, the conventional method of scanning the full spectrum of the sample and then invoking the model to analyze and predict results has a large amount of collected data, redundant information, slow acquisition speed, and high model complexity. This paper proposes a feature wavelength selection approach based on acousto-optical tunable filter (AOTF) spectroscopy and automatic machine learning (AutoML). Based on the programmable selection of sub nm center wavelengths achieved by the AOTF, it is capable of rapid acquisition of combinations of feature wavelengths of samples selected using AutoML algorithms, enabling the rapid output of target substance detection results in the field. The experimental setup was designed and application validation experiments were carried out to verify that the method could significantly reduce the number of NIR sampling points, increase the sampling speed, and improve the accuracy and predictability of NIR data models while simplifying the modelling process and broadening the application scenarios.