To address the challenges of high fine-grained similarity and background interference in recognizing teacher teaching behaviors (TTB), this paper proposes a multi-modal network, TBRNet, aiming to improve recognition performance and facilitate teaching reflection. TBRNet leverages CLIP as a semantic prior and introduces two key mechanisms: the Cascaded Collaborative Attention (CCA) module and the Dynamic Query-Driven (DQD). The CCA module performs bidirectional fusion of temporal and semantic features to capture the temporal contextual information of teaching behaviors; the DQD mechanism, using gated semantic prototypes, adaptively focuses on key discriminative regions, improving the model’s ability to distinguish subtle behavioral differences. On the specialized TBU dataset, TBRNet outperforms all baseline models, achieving a Top-1 accuracy of 86.4%. On the public benchmarks UCF-101 and HMDB-51, TBRNet achieves remarkable accuracy rates of 95.8% and 81.7%, respectively, which validates its strong generalization capability across different datasets. This study provides an effective method for efficiently identifying teacher teaching behaviors in real classroom environments.