Vision-based 3D scene perception and understanding are crucial for autonomous driving, robot navigation, and obstacle avoidance. However, describing objects with arbitrary shapes presents challenges in traditional 3D object detection tasks. In order to better perceive indoor and outdoor scenes, we propose a new method called 3DOPFormer in this paper, which can achieve 3D occupancy perception for autonomous driving through pure vision. Specifically, it first extracts multi-camera image features and uses spatial occupancy cross-attention implicit learning to lift 2D image features to 3D volume features. Then, it uses 3D deconvolution to upsample volume features. For the first time, we integrate the inherent directional and distance properties of LiDAR rays into the 3D occupancy prediction task, leading to outstanding occupancy prediction performance. And we gradually learn about dense visual occupancy perception using sparse point cloud supervision and neural rendering methods, without using expensive dense occupancy annotation supervision for training. Additionally, a cost-effective data collection scheme is introduced, and based on which, we create a dataset named ML3DOP that encompasses both indoor and outdoor scenes using 4-eye cameras and 16-beam LiDAR for 3D occupancy perception. The experimental results demonstrate that our approach can achieve accurate 3D occupancy prediction by only using multiple cameras. Dataset: https://github.com/lvchuandong/ML3DOP .

Annotations for image segmentation are expensive and time-consuming. In contrast to image segmentation, the task of object detection is in general easier in terms of the acquisition of labeled training data and the design of training models. In this paper, we combine the idea of unsupervised learning and a pretrained object-detection network to perform image segmentation, without using expensive segmentation labels. Specially, we designed a pretext task based on the sparse decomposition of object instances in videos to obtain the segmentation mask of the objects, which benefits from the sparsity of image instances and the inter-frame structure of videos. To improve the accuracy of identifying the ’right’ object, we used a pretrained object-detection network to provide the location information of the object instances, and propose an Object Location Segmentation (OLSeg) model of three branches with bounding box prior. The model is trained from videos and is able to capture the foreground, background and segmentation mask in a single image. The performance gain benefits from the sparsity of object instances (the foreground and background in our experiments) and the provided location information (bounding box prior), which work together to produce a comprehensive and robust visual representation for the input. The experimental results demonstrate that the proposed model boosts the performance effectively on various image segmentation benchmarks.

Due to the ability to perceive fine-grained 3D scenes and recognize objects of arbitrary shapes, 3D occupancy prediction plays a crucial role in vision-centric autonomous driving and robotics. However, most existing methods rely on voxel-based methods, which inevitably demand a large amount of memory and computing resources. To address this challenge and facilitate more efficient 3D occupancy prediction, we propose HBEVOcc, a Bird’s-Eye-View based method for 3D scene representation with a novel height-aware deformable attention module, which can effectively leverage latent height information within BEV framework to compensate for lack of height dimension, significantly reducing computing resource consumption while enhancing the performance. Specifically, our method first extracts multi-camera image features and lifts these 2D features into 3D BEV occupancy features via explicit and implicit view transformations. The BEV features are then further processed by a BEV feature extraction network and height-aware deformable attention module, with the final 3D occupancy prediction results obtained through a prediction head. To further enhance voxel supervision along the height axis, we introduce a height-aware voxel loss with adaptive vertical weighting. Extensive experiments on the Occ3D-nuScenes and OpenOcc dataset demonstrate that HBEVOcc can achieve state-of-the-art results in terms of both mIoU and RayIoU metrics with less training memory (even when trained on 2080Ti).

Image anomaly detection has emerged as a crucial field in data analysis, pivotal in identifying unusual patterns in vast visualizations, with implications across numerous industries and academic research. Despite its importance, image anomaly detection faces significant challenges, primarily due to the limited availability of anomaly samples that hinder sufficient supervised training. Current methods also suffer from limitations such as reliance on complex network architectures, the necessity of task-specific loss functions, and a tendency for over-generalization, which can compromise detection accuracy. To address these issues, we propose the Pre-trained Feature Extraction for Activation Learning (PFEAL), a novel framework in image anomaly detection that combines a pre-trained feature extractor with an activation learning-based neural network. The PFEAL framework is distinctively designed to operate without relying on complicated model architectures or task-specific loss functions, streamlining the process of anomaly detection. It leverages activation values to reflect the abnormality of input patterns, providing a straightforward, effective, and interpretable anomaly detection framework. Extensive experiments validate the efficacy of PFEAL, demonstrating superior performance compared to baselines and state-of-the-art methods. Moreover, its capabilities for few-shot learning and its intuitive new paradigm offer a flexible and practical approach to image anomaly detection.

In photonics research, neural networks have been proven to be efficient and accurate in forward modeling and inverse design of photonic components. However, when solving dynamic electromagnetic fields, most deep learning models require a large amount of training data, rarely consider the three variables of material, geometry, and source simultaneously, or calculate fields step by step. To address these issues, we propose an unsupervised time-domain MaxwellNet based on physical information for quickly and accurately predicting the dynamic electromagnetic fields of structures with variable materials and geometries under different sources. Depending on whether the source parameters change during training, we propose two basic models, Multi-step Net and S-Multi-step Net, and suggest classifying the source parameters and adding ResBlock modules to handle the case where the source parameters are variable and the excitation value is non-zero and changes over time. Furthermore, we propose a looping strategy, which significantly improves the time extrapolation performance by several loops that are far less than the time steps of the electromagnetic fields to be solved. The governing equations are embedded into the loss function through finite differences to guide the training. We demonstrate the effectiveness and advantages of the proposed method through two sets of numerical experiments with different source. The trained network can predict samples that differ greatly from the training data without fine-tuning. Finally, we incorporate the time-domain MaxwellNet as a simulator into an inverse model and show that this forward model works well for solving the inverse problem.