Neural Radiance Fields (NeRFs) have revolutionized 3D scene synthesis. Voxel grids are commonly employed to enhance training or rendering speed, but they entail additional storage requirements. The large model size and high computational and memory demands impede their progress on resource-constrained devices, e.g., Microcontroller Units (MCUs). Besides, there is currently no NeRF rendering framework available on MCU devices. In this paper, we propose a NeRF method named MCUNeRF for 3D scene synthesis on MCU devices. The proposed MCUNeRF compresses voxel grids via a hybrid quantization algorithm merging learned step-size quantization (LSQ) and optimized product quantization (OPQ). To further reduce the model storage, we also propose a codebook-sharing method that renders multiple objects with a single quantization codebook. Then we implement a NeRF-based rendering framework for MCU devices, which leverages a low-bit neural network computation framework, i.e. CMSIS-NN, to accelerate the rendering progress. Extensive experiments on four datasets such as Synthetic-NeRF demonstrate that our proposed method could compress model data by 20-40 times with comparable rendering quality, which enables NeRF-based scene rendering on MCU devices with only 1M SRAM.
13 August 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Heatmap-based methods have dominated the face alignment task, yet the maximum response decoding scheme necessitates further reform. While some studies have attempted to compensate for prediction offsets using a post-processing module, the prediction errors induced by the maximum response decoding scheme remain challenging to rectify. In this paper, we assume that using heatmap value to denote the ground-truth probability is not accurate enough. To cure this problem, we propose DISPAL, a novel DIS tribution-based P robability for f A cial L andmarks, which signifies the ground-truth probability by the similarity between the pixel’s neighbouring value distribution and Gaussian distribution. This innovative probability enables us to pinpoint the keypoint location more robustly than previous methods that rely solely on the peak score. It also exhibits remarkable generalization to complex decoding methodologies. Furthermore, we propose supervising this probability as an additional task loss to help the model learn better heatmap representation. Extensive empirical results on WFLW, 300W, and COFW datasets demonstrate that our distribution-based probability mechanism significantly surpasses original value-based probability approaches.
Vision-Language-Action (VLA) models extend vision-language models to embodied control by mapping natural-language instructions and visual observations to robot actions. Despite their capabilities, VLA systems face significant challenges due to their massive computational and memory demands, which conflict with the constraints of edge platforms such as on-board mobile manipulators that require real-time performance. Addressing this tension has become a central focus of recent research. In light of the growing efforts toward more efficient and scalable VLA systems, this survey provides a systematic review of approaches for improving VLA efficiency, with an emphasis on reducing latency, memory footprint, and training and inference costs. We categorize existing solutions into four dimensions: model architecture, perception feature, action generation, and training/inference strategies, summarizing representative techniques within each category. Finally, we discuss future trends and open challenges, highlighting directions for advancing efficient embodied intelligence.
Fully convolutional detectors discard the one-to-many assignment and adopt a one-to-one assigning strategy to achieve end-to-end detection but suffer from the slow convergence issue. In this paper, we revisit these two assignment methods and find that bringing one-to-many assignment back to end-to-end fully convolutional detectors helps with model convergence. Based on this observation, we propose {\em \textbf{D}ual \textbf{A}ssignment} for end-to-end fully convolutional de\textbf{TE}ction (DATE). Our method constructs two branches with one-to-many and one-to-one assignment during training and speeds up the convergence of the one-to-one assignment branch by providing more supervision signals. DATE only uses the branch with the one-to-one matching strategy for model inference, which doesn't bring inference overhead. Experimental results show that Dual Assignment gives nontrivial improvements and speeds up model convergence upon OneNet and DeFCN. Code: https://github.com/YiqunChen1999/date.