The plant-based dietary pattern has been recommended for its potential health and environmental benefits, but its association with bone loss needs to be further explored. This study aimed to investigate the association between three plant-based diet indexes and bone loss in 16,085 adults, using data from the National Health and Nutrition Examination Survey. Three plant-based diet indexes (PDI, hPDI, and uPDI) were calculated from two NHANES 24-h dietary recall interviews, to characterize a plant-based diet. A multinomial logistic regression model was used to estimate the odds ratios (OR) and 95% confidence intervals (95% CI). Higher hPDI and PDI were associated with increased risk of bone loss (OR_{Q5 vs. Q1} = 1.50; 95% CI: 1.24–1.81 for hPDI; OR_{Q5 vs. Q1} = 1.22; 95% CI: 1.03–1.45 for PDI), while higher uPDI was associated with increased risk of osteoporosis (OR_{Q5 vs. Q1} = 1.48; 95% CI: 1.04–2.11). A harmful association between plant-based diet indexes (hPDI and PDI) and osteopenia was observed at the lumbar spine rather than the femoral neck. We conducted several sensitivity analyses to ensure the robustness of results, including subgroup analysis, exclusion of people taking anti-osteoporotic and estrogenic drugs, further adjustment for menopausal status, corticosteroid usage, and dietary supplements, and calculation of E-value. Our study demonstrates the deleterious effects of a plant-based diet on bone health and emphasizes the importance of a balanced diet.