Applications for millimeter-wave (mmWave) radars have become increasingly popular in human activity recognition. Many researchers have combined radars with neural networks and gained a high performance on various applications. However, most of these studies feed the raw point cloud data directly into the networks, which can be unstable and inaccurate under certain circumstances. In this paper, we define a reliability measure of the point cloud data and design a novel voxelization algorithm to reconstruct the data. Experiments show that our algorithm can improve the stability of the point cloud generated from mmWave radars in terms of error reduction and scene re-construction. We demonstrate the effectiveness of our proposed algorithm using a neural network-based system for identifying a person’s sitting direction. In our experiment, compared with the baseline, our voxelization algorithm can improve the system in terms of accuracy (4.3%), training time (55.6%), and computational complexity, which is more suitable for light-weighted networks and low energy consumption platforms.