An accurate and fast building load prediction model is critically important for guiding building energy system design, optimizing operational parameters, and balancing a power grid between energy supply and demand. A physics-based simulation tool is traditionally used to provide the building load demand; however, it is constrained by its complex model development process and requirement for engineering judgments. Machine learning algorithms (i.e., data-driven models) based on big data can bridge this gap. In this study, we used the massive energy data generated by a physics-based tool (EnergyPlus) to develop three data-driven models (i.e., LightGBM, random forest (RF), and long-short term memory (LSTM)) and compared their prediction performances. The physics-based models were developed using office prototype building models as baselines, and ranges were provided for selected key input parameters. Three different input feature dimensions (i.e., six-, nine-, and fifteen-input feature selections) were investigated, aiming to meet different demands for practical applications. We found that LightGBM significantly outperforms the RF and LSTM algorithms, not only with respect to prediction accuracy but also in regard to computation cost. The best prediction results show that the coefficient of variation of the root mean squared error (CVRMSE), squared correction coefficient (R²), and computation time are 5.25%, 0.9959, and 7.0 s for LightGBM, respectively, evidently better than the values for the algorithms based on RF (18.54%, 0.9482, and 44.6 s) and LSTM (22.06%, 0.9267, and 758.8 s). The findings demonstrate that a data-driven model is able to avoid the process of establishing a complicated physics-based model for predicting a building’s thermal load, with similar accuracy to that of a physics-based simulation tool.