Voxel-based neural networks have been widely applied to the processing of large-scale outdoor point cloud data, which first convert points into voxels and then extract features using several sparse convolution and normal convolution layers. The hardware implementation of these networks suffers from complex rulebook generation, irregular memory access, and low hardware utilization. Meanwhile, these networks still have much data sparsity. In this paper, we propose an energy and area-efficient query-based sparsity-aware accelerator for voxel-based point cloud neural networks, namely QSAP. Specifically, a dedicated unit is used to improve the efficiency of rulebook generation. A query-based input feature-writing method is proposed to enhance parallel reading potential. An efficient weight-mapping method is introduced to store unpruned weights in on-chip buffers. A novel input-feature-reading method with consecutive queries is proposed to enable out-of-order execution of convolution operations, thereby improving hardware utilization. The hardware utilization is further enhanced by a proposed pop strategy that minimizes the total number of empty FIFOs. The MAC related to zero value is also skipped in this process. As a result, QSAP achieves superior performance on 22 nm technology with the throughput, energy efficiency, area efficiency, frame rate, and frame energy of 1074 GOPS, 7.79 TOPS/W, 590 GOPS/mm $\mathbf {^{2}}$ , 35.2 FPS, and 3.91 mJ/Frame, respectively, better than state-of-the-art works.