With the prosperity of the intelligent surveillance, multiple cameras have been applied to localize pedestrians more accurately. However, previous methods rely on laborious annotations of pedestrians in every frame and camera view. Therefore, we propose in this paper an Unsupervised Multi-view Pedestrian Detection approach (UMPD) to learn an annotation-free detector via vision-language models and 2D-3D cross-modal mapping: 1) Firstly, Semantic-aware Iterative Segmentation (SIS) is proposed to extract unsupervised representations of multi-view images, which are converted into 2D masks as pseudo labels, via our proposed iterative PCA and zero-shot semantic classes from vision-language models; 2) Secondly, we propose Geometry-aware Volume-based Detector (GVD) to end-to-end encode multi-view 2D images into a 3D volume to predict voxel-wise density and color via 2D-to-3D geometric projection, trained by 3D-to-2D rendering losses with SIS pseudo labels; 3) Thirdly, for better detection results, i.e., the 3D density projected on Birds-Eye-View, we propose Vertical-aware BEV Regularization (VBR) to constrain pedestrians to be vertical like the natural poses. Extensive experiments on popular multi-view pedestrian detection benchmarks Wildtrack, Terrace, and MultiviewX, show that our proposed UMPD, as the first fully-unsupervised method to our best knowledge, performs competitively to the previous state-of-the-art supervised methods. Code is available at https://github.com/lmy98129/UMPD.

Open-vocabulary object detection (OVD) models are considered to be Large Multi-modal Models (LMM), due to their extensive training data and a large number of parameters. Mainstream OVD models prioritize object coarse-grained category rather than focus on their fine-grained attributes, e.g., colors or materials, thus failed to identify objects specified with certain attributes. Despite being pretrained on large-scale image-text pairs with rich attribute information, their latent feature space does not highlight these fine-grained attributes. In this paper, we introduce HA-FGOVD, a universal and explicit method that enhances the attribute-level detection capabilities of frozen OVD models by highlighting fine-grained attributes in explicit linear space. Our approach uses a LLM to extract attribute words in input text as a zero-shot task. Then, token attention masks are adjusted to guide text encoders in extracting both global and attribute-specific features, which are explicitly composited as two vectors in linear space to form a new attribute-highlighted feature for detection tasks. The composition weight scalars can be learned or transferred across different OVD models, showcasing the universality of our method. Experimental results show that HA-FGOVD achieves state-of-the-art performance on the FG-OVD benchmark and demonstrates promising generalization on the OVDEval benchmark, suggesting that our method addresses significant limitations in fine-grained attribute detection and has potential for broader fine-grained detection applications.

Text-based person retrieval (TBPR) aims to search for target person images from large-scale video clips or image databases based on textual descriptions. The quality of benchmarks is critical to accurately evaluating TBPR models for their ability in relation to cross-modal matching. However, we find that existing TBPR benchmarks have a common problem, which often leads to ambiguities where multiple images of persons with different identities have very similar or even identical textual descriptions. As a consequence, although TBPR models correctly retrieve the images corresponding to a given description, such matches may be erroneously evaluated as mismatches due to the above annotation problem. We argue that the main cause of this problem is that each person image is annotated individually without reference to other similar images, making it challenging to provide distinctive descriptions for each image. To address this problem, we propose an effective and efficient annotation refinement framework to improve the annotation quality of TBPR benchmarks and thereby mitigate annotation-induced mismatches. Firstly, sets of images prone to mismatches are automatically identified by TBPR models. Then, by leveraging multimodal large language models (MLLMs), multiple images are simultaneously processed and distinctive descriptions are generated for each image. Finally, the original descriptions are replaced to improve the annotation quality. Extensive experiments on three popular TBPR benchmarks (CUHK-PEDES, RSTPReid and ICFG-PEDES) validate the effectiveness of our proposed method for improving the quality of annotations, and demonstrate that the resulting more discriminative captions can truly benefit the mainstream TBPR models. The improved annotations of these benchmarks will be released publicly.

Oxygen lance control is a critical decision-making task in basic oxygen furnace (BOF) steelmaking. It regulates lance height to control decarburization rate, aiming to achieve stability. Existing approaches either rely on human expertise, requiring long-term training and experience accumulation, or use traditional machine-learning models that simplify the problem into numerical regression, resulting in limited interpretability and adaptability. In this work, we are the first to reformulate oxygen lance control as a structured multimodal generative decision-making task and propose a self-evolving reasoning Multimodal Large Language Model (MLLM) agent that performs structured reasoning. The agent jointly perceives real-time furnace mouth flame videos and historical process data to generate structured decision paths. To address the scarcity of high-quality annotated industrial data, we introduce OT-SERA, an Operation‐Trajectory guided Self‐Evolving Reasoning MLLM Agent for structured decision-making. OT-SERA includes an initialization phase and a refinement phase. In the Operation-Trajectory Guided Reasoning Initialization (OTGRI) phase: operation trajectories from real BOF steelmaking guide the generation of structured chain-of-thoughts (CoTs), which are then used to warm-up the agent by fine-tuning. In the Self-Evolving Reasoning Refinement (SERR) phase: the agent improves reasoning paths through self-evolving updates with path backtracking, automatically constructing high-quality CoTs without additional human annotation. Repeated the SERR further improve CoTs quality, highlighting the effectiveness. Extensive experiments on real-world BOF data show that OT-SERA outperforms strong time series and multimodal baselines under both normal and hard operating conditions. By integrating multimodal perception, OTGRI, and SERR, this work provides an effective solution for intelligent control of complex industrial processes.

With the prosperity of e-commerce applications, the web data of products are presented by multiple modalities, e.g., vision and language. For mining the product characteristics, multimodal attribute values are crucial, which are extracted from textual descriptions, assisted by helpful image regions. However, most previous works (1) fuse the multimodal information within a newly learned range based on co-occurrence rather than language meanings and (2) predict the outputs within a range of all attributes rather than the product-related ones. These issues yield unsatisfactory results; thus, we propose a novel approach via Dynamic Range Modulation (DRAM): (1) First, we propose an Information Range Calibration (IRC) method to dynamically fuse multimodal features of related meanings as Text-Related Embeddings (TEM) within a language range, which is calibrated from the range to fuse language features by a powerful attention mechanism of a pretrained language model. (2) Moreover, an Attribute Range Minimization (ARM) method is proposed to minimize the output attribute range based on the adaptive selection of product-related attribute prototypes. Experiments on the popular multimodal e-commerce benchmarks show that our DRAM performs well compared with previous methods.