6 March 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
In this paper, we propose a scene-level inverse rendering framework that uses multi-view images to decompose the scene into geometry, SVBRDF, and 3D spatially-varying lighting. While multi-view images have been widely used for object-level inverse rendering, scene-level inverse rendering has primarily been studied using single-view images due to the lack of a dataset containing high dynamic range multi-view images with ground-truth geometry, material, and spatially-varying lighting. To improve the quality of scene-level inverse rendering, a novel framework called Multi-view Attention Inverse Rendering (MAIR) was recently introduced. MAIR performs scene-level multi-view inverse rendering by expanding the OpenRooms dataset, designing efficient pipelines to handle multi-view images, and splitting spatially-varying lighting. Although MAIR showed impressive results, its lighting representation is fixed to spherical Gaussians, which limits its ability to render images realistically. Consequently, MAIR cannot be directly used in applications such as material editing. Moreover, its multi-view aggregation networks have difficulties extracting rich features because they only focus on the mean and variance between multi-view features. In this paper, we propose its extended version, called MAIR++. MAIR++ addresses the aforementioned limitations by introducing an implicit lighting representation that accurately captures the lighting conditions of an image while facilitating realistic rendering. Furthermore, we design a directional attention-based multi-view aggregation network to infer more intricate relationships between views. Experimental results show that MAIR++ not only outperforms MAIR and single-view-based methods but also demonstrates robust performance on unseen real-world scenes.
22 December 2021 in IEEE Access
There are fundamental difficulties in obtaining material type of an arbitrary object using traditional sensors. Existing material type recognition methods mostly focus on color based visual features and object-prior. Surface reflectance is another critical clue in the characterization of certain material type and can be observed by traditional sensors such as color camera and time-of-flight depth sensor. A material type is characterized well by relevant surface reflectance together with traditional visual appearance providing better description for material type recognition. In this work, we propose a material type recognition method based on both color and reflectance features using deep neural network. Proposed method is evaluated on both public and our own data sets showing promising material type recognition results.
17 September 2025 in IEEE Access
Neural Radiance Fields (NeRF) has demonstrated its efficacy in synthesizing novel view images, and has yielded promising outcomes under specific conditions: 1) dense, multi-view observations, 2) constrained illumination across input views, and 3) known camera poses. In real-world scenarios, however, these conditions are rarely satisfied. Especially when confronted with a limited number of input views and varying illumination conditions, NeRF experiences diminished performance. While some of the previous works have addressed novel view synthesis under unconstrained illumination, their effectiveness still hinges on dense observations. In this paper, we tackle the challenge of few-shot NeRF under unconstrained illumination by leveraging illumination-invariant intrinsic components to provide stronger cross-view regularization. To further enhance practicality, we design a lightweight extension that achieves comparable improvements with minimal computational overhead. We also establish new benchmarks featuring diverse viewpoints and illumination conditions, and demonstrate through extensive experiments that our framework consistently improves robustness and synthesis quality in real-world settings.