Origin–destination (OD) flow prediction is fundamental to intelligent transportation systems, yet existing diffusion-based models face two critical limitations. First, they inadequately exploit spatial semantics, focusing primarily on temporal dependencies or topological correlations while neglecting urban functional heterogeneity encoded in Points of Interest (POIs). Second, static embedding fusion cannot dynamically capture semantic importance variations during denoising—particularly during traffic surges in POI-dense areas. To address these gaps, we propose the Cross-Attention Diffusion Model (CADM), a semantically conditioned framework for short-term OD flow forecasting. CADM integrates POI embeddings as spatial semantic priors and employs cross-attention to enable semantic-guided denoising, facilitating dynamic spatiotemporal feature fusion. This design adaptively reweights regional representations throughout reverse diffusion, enhancing the model’s capacity to capture complex mobility patterns. Experiments on real-world datasets demonstrate that CADM achieves balanced performance across multiple metrics. At the 30 min horizon, CADM attains the lowest RMSE of 5.77, outperforming iTransformer by 1.9%, while maintaining competitive performance at the 15 min horizon. Ablation studies confirm that removing POI features increases prediction errors by 15–20%, validating the critical role of semantic conditioning. These findings advance semantic-aware generative modeling for spatiotemporal prediction and provide practical insights for intelligent transportation systems, particularly for newly established transportation hubs or functional zone reconfigurations where semantic understanding is essential.