Mycoplasma pneumoniae (M. pneumoniae) is one of the major causes of community-acquired pneumonia, accounting for 20–40% of total cases. Rapid and accurate detection of M. pneumoniae is crucial for the diagnosis and rational selection of antibiotics. In this study, we set up a real-time recombinase polymerase amplification (RPA) assay to detect the conserved gene CARDS of M. pneumoniae. The amplification can be finished in 20 min at a wide temperature range from 37–41 °C. The limit of detection of RPA assay was 10 fg per microliter. Cross-reaction with commonly detected respiratory pathogens was not observed using RPA assay. Among clinical sputum samples, the detection rate of RPA assay and real-time PCR assay was 48.4% (92/190) and 46.3% (88/190), respectively (p = 0.68). Therefore, the RPA assay for M. pneumoniae detection is rapid and easy to use and may serve as a promising test for early diagnosis of M. pneumoniae infection.