The evolution of 3D visualization techniques has fundamentally transformed how we interact with digital content. At the forefront of this change is point cloud technology, offering an immersive experience that surpasses traditional 2D representations. However, the massive data size of point clouds presents significant challenges in data compression. Current methods for lossy point cloud attribute compression (PCAC) generally focus on reconstructing the original point clouds with minimal error. However, for point cloud visualization scenarios, the reconstructed point clouds with distortion still need to undergo a complex rendering process, which affects the final user-perceived quality. In this paper, we propose an end-to-end deep learning framework that seamlessly integrates PCAC with differentiable rendering, denoted as rendering-oriented PCAC (RO-PCAC), directly targeting the quality of rendered multiview images for viewing. In a differentiable manner, the impact of the rendering process on the reconstructed point clouds is taken into account. Moreover, we characterize point clouds as sparse tensors and propose a sparse tensor-based transformer, called SP-Trans. By aligning with the local density of the point cloud and utilizing an enhanced local attention mechanism, SP-Trans captures the intricate relationships within the point cloud, further improving feature analysis and synthesis within the framework. Extensive experiments demonstrate that the proposed RO-PCAC achieves state-of-the-art compression performance, compared to existing reconstruction-oriented methods, including traditional, learning-based, and hybrid methods. The code will be released at https://github.com/net-F/RO-PCAC.git.

Micro-lens array (MLA) based light field cameras have been commercialized and had a wide range of applications in various fields in recent years. To promote these applications, especially for 3D reconstruction, it is crucial to calibrate light field cameras. Among the calibration steps, it is of great importance to detect corners in checkerboard images and establish the point-to-point correspondence between the checkerboard and its images. However, almost all existing algorithms detect corners in the sub-aperture images instead of raw images for conventional light field cameras. In this paper, a corner detection algorithm that can recognize corners in the raw images of conventional light field cameras was proposed. Firstly, template matching was used to detect the 3D lines in the image space of the main lens. Then, the locations of the corners were obtained by calculating the intersection of 3D line segments and reprojecting it to the raw image again. Experimental results have demonstrated that the algorithm can still achieve good performance in actual datasets with blurred and low resolution micro lens image.

Point clouds are widely used as representations of 3D objects and scenes in a number of applications, including virtual and mixed reality, autonomous driving, antiques reconstruction. To reduce the cost for transmitting and storing such data, this paper proposes an end-to-end learning-based point cloud attribute compression (PCAC) approach. The proposed network adopts a sparse convolution-based variational autoencoder (VAE) structure to compress the color attribute of point clouds. Considering the difficulty of stacked convolution operations in capturing long range dependencies, the attention mechanism is incorporated in which a non-local attention module is developed to capture the local and global correlations in both spatial and channel dimensions. Towards the practical application, an additional modulation network is offered to achieve the variable rate compression purpose in a single network, avoiding the memory cost of storing multiple networks for multiple bitrates. Our proposed method achieves state-of-the-art compression performance compared to other existing learning-based methods and further reduces the gap with the latest MPEG G-PCC reference software TMC13 version 14.

The resources of edge devices in AIoT systems are usually constrained with size and power. The computational complexity of neural network models in these edge devices has become a major concern. The most compact form of deep neural networks is binarized neural network (BNN), which adopts binary weights and exclusive NOR (XNOR) operations as binary convolution. In this paper, we propose weight compression-friendly BNN to save hardware resources by reducing memory space. The proposed technique does not binarize weights just according to the signs of weights, but fully considers compression efficiency in the training of the BNN model. The experiments are performed by using the binary version of a 6-layer convolutional neural network (CNN) and MNIST case. The results show that the proposed technique can achieve more than 25% reduction in memory space with accuracy loss of 1 %, or more than 35% memory reduction with about 2.5% accuracy drop for MNIST classification. The weight compression method does not destroy the regular structure of neural networks, so the proposed technique is very fit for processor-based BNN hardware accelerators.

Hand-held light field (LF) cameras have unique advantages in computer vision such as 3D scene reconstruction and depth estimation. However, the related applications are limited by the ultra-small baseline, leading to the extremely low depth resolution in reconstruction. To solve this problem, we propose to rectify LF to obtain a large baseline. Specifically, the proposed method aligns two LFs captured by two hand-held LF cameras with a random relative pose, and extracts the corresponding row-aligned sub-aperture images (SAIs) to obtain an LF with a large baseline. For an accurate rectification, a method for pose estimation is also proposed, where the relative rotation and translation between the two LF cameras are estimated. The proposed pose estimation minimizes the degree of freedom (DoF) in the LF-point-LF-point correspondence model and explicitly solves this model in a linear way. The proposed pose estimation outperforms the state-of-the-art algorithms by providing more accurate results to support rectification. The row-aligned SAIs and significantly improved depth resolution in 3D reconstruction demonstrate the effectiveness of the proposed LF rectification.

Accurately calibrating an unfocused plenoptic camera is essential to its applications. Rapid progress has been made in this area in the past decades. In this paper, detailed analysis is first performed toward the state-of-the-art projection model. Based on the analysis, parameters in the projection model are divided into two groups. Then, based on the parameter analysis, a new, to the best of our knowledge, form of the projection model, together with a new image feature light field structure point (LF-structure-point), is proposed. The LF-structure-point provides a completely non-redundant representation of the signal structure of the recorded light field raw data and induces a virtual space, “light field structure space,” which is related to the real physical space by a 3D-to-3D projective transformation. The extracting algorithm of the LF-structure-point is also presented. Finally, based on the 3D-to-3D projective transformation and parameter grouping, a simple two-step calibration method using the LF-structure-point as the input data is then proposed and achieves satisfactory experimental results.