Auxin response factors (ARFs) are a class of significant transcription factors that play crucial roles in the regulation of plant growth and development. Although ARF genes have been extensively characterized in various plants, their functions in perennial woody crops, particularly their involvement in regulating starch accumulation—a key determinant of yield and quality in Castanea mollissima—remain largely unexplored. To address this knowledge gap, we conducted a comprehensive study of the ARF gene family in the C. mollissima. In this study, 18 CmARF members, exhibiting diverse physicochemical properties, were identified within the C. mollissima genome. These CmARFs were categorized into four groups. Dispersed duplication emerged as the primary mechanism driving the expansion of the CmARF gene family. As C. mollissima seed kernels developed, notable changes were observed in starch content and the activity of enzymes related to starch biosynthesis, particularly a significant decrease in GBSS activity, which corresponded with an increase in seed kernel size and starch content. Transcriptome analysis delineated the expression patterns of CmARF genes during the development of C. mollissima seed kernels. A key novel finding of our research is that CmARF5a and CmARF18 are hypothesized to act as pivotal repressors of starch accumulation. This hypothesis is based on their expression profiles, strong negative correlations with physiological indicators, and WGCNA. Notably, the lack of correlation between these CmARFs and the expression of core starch biosynthetic genes suggests a potential indirect regulatory mechanism, offering a new perspective on ARF function in storage organ development. This study not only provides the first comprehensive characterization of the CmARF family but also offers a theoretical framework and candidate genes for future functional research on C. mollissima seed kernel development and starch biosynthesis.