Tapiscia sinensis Oliv. (Tapisciaceae) has been proven to be a functional androdioecious species with both male and hermaphroditic individuals, and the pollen viability of males is far higher than that of hermaphrodites. To better understand the causes of the low pollen viability in hermaphroditic flowers, different stages of anther development were observed. We found that hermaphroditic flowers exhibit abnormal tapetum development, resulting in low pollen viability. To clarify the underlying molecular mechanism of abnormal tapetum development in hermaphrodites, quantitative real-time PCR analyses were performed. The results revealed that the expression levels of an important transcription factor for tapetum development and function, T. sinensis DYSFUNCTIONAL TAPETUM1 (TsDYT1), and its potential downstream regulatory genes T. sinensis DEFECTIVE in TAPETAL DEVELOPMENT and FUNCTION1 (TsTDF1), T. sinensis ABORTED MICROSPORE (TsAMS) and T. sinensis MALE STERILITY 1 (TsMS1) were all significantly downregulated in hermaphrodites compared with males at some key stages of anther development. The amino acid sequence similarity, expression pattern, gene structure and subcellular localization of these genes were analyzed, and the results indicated functional conservation between T. sinensis and homologues in Arabidopsis thaliana. Next, rapid amplification of cDNA end and thermal asymmetric interlaced PCR were employed to clone the full-length cDNA and promoter sequences of these genes, respectively. In addition, results of yeast two-hybrid analysis showed that TsDYT1 can form heterodimers with TsAMS, and yeast one-hybrid analysis demonstrated that TsDYT1 directly binds to the promoter regions of TsTDF1 and TsMS1. TsTDF1 can directly regulate expression of TsAMS, suggesting that a functionally conserved pathway exists between A. thaliana and T. sinensis to regulate tapetum development. In conclusion, the results suggest that abnormal expression of core transcription factors for tapetum development, including TsDYT1, TsTDF1, TsAMS and TsMS1, plays an important role in the abnormal development of the tapetum in T. sinensis hermaphrodites. Furthermore, a hermaphroditic tapetum with abnormal function causes the low pollen viability of hermaphroditic trees. Our results provide new insight into our understanding of the underlying mechanism of why pollen viability is much higher in males than hermaphrodites of the androdioecious tree T. sinensis.