Many methods have been used to generate gridded population maps by downscaling demographic data. As one of these methods, the accuracy of the dasymetric model depends heavily on the covariates. Point-of-interest (POI) data, as important covariates, have been widely used for population estimation. However, POIs are often used indiscriminately in existing studies. A few studies further used selected categories of POIs identified based only on the nonspatial quantitative relationship between the POIs and population. In this paper, the spatial association between the POIs and population distribution was considered to identify the POIs with a strong spatial correlation with the population distribution, i.e., population-sensitive POIs. The ability of population-sensitive POIs to improve the fine-grained population mapping accuracy was explored by comparing the results of random forest dasymetric models driven by population-sensitive POIs, all POIs, and no POIs, along with the same sets of multisource remote sensing and social sensing data. The results showed that the model driven by population-sensitive POI had the highest accuracy. Population-sensitive POIs were also more effective in improving the population mapping accuracy than were POIs selected based only on their quantitative relationship with the population. The model built using population-sensitive POIs also performed better than the two popular gridded population datasets WorldPop and LandScan. The model we proposed in this study can be used to generate accurate spatial population distribution information and contributes to achieving more reliable analyses of population-related social problems.