A suitable self‐thinning model is fundamental to effective density control and management. Using data from 265 plot measurements in oak mixed forests in central China, we demonstrated how to estimate a suitable self‐thinning line for oak mixed forests from three aspects, i.e., self‐thinning models (Reineke's model and the variable density model), statistical methods (quantile regression and stochastic frontier analysis), and the variables affecting stands (topography and stand structure factors). The proposed variable density model, which is based on the quadratic mean diameter and dominant height, exhibited a better goodness of fit and biological relevance than Reineke's model for modeling the self‐thinning line for mixed oak forests. In addition, the normal‐truncated normal stochastic frontier model was superior to quantile regression for modeling the self‐thinning line. The altitude, Simpson index, and dominant height–diameter ratio (/D) also had significant effects on the density of mixed forests. Overall, a variable density self‐thinning model may be constructed using stochastic frontier analysis for oak mixed forests while considering the effects of site quality and stand structure on density. The findings may contribute to a more accurate density management map for mixed forests.

Stand density management is important for decision-making regarding adaptive silviculture and thinning, growth modelling, and yield prediction in forests, especially plantations. Although substantial research related to the self-thinning rule and maximum size-density law has been conducted, there are still critical gaps that exist in the biophysical explanation and validation of the relationships among stand variables and relevant parameters. In this study, time series observations from six plots of fully stocked Chinese fir plantations with different densities of planted trees were used to characterise the growth of stand basal area (G), average height (H), and diameter at breast height (D). The growth trends in the stand parameters and the relationships among them were analysed. As indicated by previous studies, in the fully stocked stands, there was a significant linear relationship between G and H. This study also resulted in the following new findings: (1) At the beginning, the growth rate of stand basal area ($P_G$) was greater than the growth rate of average height ($P_H$), but $P_G$ decreased quickly as the stands approached canopy closure and then became stable. Meanwhile, as the stands neared canopy closure, the rate of increase in the G/H ratio decelerated, ultimately resulting in a stable G/H value that approached the first limit value. This led to a stand growth balance status that continued until self-thinning took place. (2) Artificial thinning broke the growth balance status, but the stands returned to balance status if they were still young enough. Self-thinning also broke the growth balance status and lead to fluctuating growth rates of both G and H, but the fluctuations were very slight, which showed a trend in similar growth rates of G and H. (3) The findings implied that the stand G and H growths were allometric at the beginning but became isogonic as canopy closure and self-thinning were approached. On the other hand, the H growth rate was generally greater than that of D, but both growth rates showed a trend in similar values after the stands matured. Subsequently, the H/D ratio is anticipated to stabilize and gradually converge towards the second limit value once the stands reach maturity. The results implied that the stand growth balance status and two limit values can be used to identify and select fully stocked stands that are needed for the development of maximum size-density equations and self-thinning rules.