Dendrobium officinale is a rare and traditional medicinal plant with high pharmacological and nutritional value. The self-incompatibility mechanism of D. officinale reproductive isolation was formed in the long-term evolution process, but intraspecific hybridization of different germplasm resources leads to a large gap in the yield, quality, and medicinal value of D. officinale. To investigate the biological mechanism of self-incompatibility in D. officinale, cytological observation and the transcriptome analysis was carried out on the samples of self-pollination and cross-pollination in D. officinale. Results for self-pollination showed that the pollen tubes could grow in the style at 2 h, but most of pollen tubes stopped growing at 4 h, while a large number of cross-pollinated pollen tubes grew along the placental space to the base of ovary, indicating that the self-incompatibility of D. officinale may be gametophyte self-incompatibility. A total of 63.41 G basesum of D. officinale style samples from non-pollinated, self-pollination, and cross-pollination by RNA-seq were obtained, and a total of 1944, 1758, and 475 differentially expressed genes (DEGs) in the comparison of CK (non-pollinated) vs. HF (cross-pollination sample), CK vs. SF (self-pollination sample) and SF vs. HF were identified, respectively. Forty-one candidate genes related to self-incompatibility were found by function annotation of DEGs, including 6 Ca²⁺ signal genes, 4 armed repeat containing (ARC) related genes, 11 S-locus receptor kinase (SRK) related genes, 2 Exo70 family genes, 9 ubiquitin related genes, 1 fatty acid related gene, 6 amino acid-related genes, 1 pollen-specific leucine-rich repeat extensin-like protein (LRX) related gene and 1 lectin receptor-like kinases (RLKs) related gene, showed that self-incompatibility mechanism of D. officinale involves the interaction of multiple genes and pathways. The results can provide a basis for the study of the self-incompatibility mechanism of D. officinale, and provide ideas for the preservation and utilization of high-quality resources of D. officinale.