Tree peony seeds, traditionally used for edible oil production, are rich in α-linolenic acid (ALA). However, little attention is paid to their development as a healthcare food due to their bitter and astringent taste. The aim of this study was to optimize the debittering process of peony seeds on the basis of maintaining nutritional value and to identify the compounds that cause the taste of bitterness. We first optimized the debittering process by orthogonal experiments which reduced the polyphenol content by 90.25%, and we measured the main nutritional value of fatty acid composition, indicating that the high content of ALA is not affected by debittering. Then, we identified and determined the types and content of polyphenols, the metabolites causing bitter taste, in the samples based on LC-ESI-QQQ-MS. Principal component analysis (PCA) and orthogonal partial least square discriminant analysis (OPLS-DA) were used to compare and analyze the seeds at different stages of debittering. Thirty-eight key metabolites were identified, of which paeoniflorin, taxifolin, alibiflorin, protocatechuic acid, benzoyl paeoniflorin, quercetin-3-galactoside, and oxpaeoniflorin were significantly compared, and most of them were positively correlated with bitter taste. These results are conducive to the exploration and study of the bitter taste and nutritional value of tree peony seeds in the future.