Forests are the main part of the terrestrial ecosystem. Airborne LiDAR is fast, comprehensive, penetrating, and contactless and can depict 3D canopy information with a high efficiency and accuracy. Therefore, it plays an important role in forest ecological protection, tree species recognition, carbon sink calculation, etc. Accurate recognition of individual trees in forests is a key step to various application. In real practice, however, the accuracy of individual tree segmentation (ITS) is often compromised by under-segmentation due to the diverse species, obstruction and understory trees typical of a high-density multistoried mixed forest area. Therefore, this paper proposes an ITS optimization method based on Gaussian mixture model for airborne LiDAR data. First, the mean shift (MS) algorithm is used for the initial ITS of the pre-processed airborne LiDAR data. Next, under-segmented samples are extracted by integrated learning, normally segmented samples are classified by morphological approximation, and the approximate distribution uncertainty of the normal samples is described with a covariance matrix. Finally, the class composition among the under-segmented samples is determined, and the under-segmented samples are re-segmented using Gaussian mixture model (GMM) clustering, in light of the optimal covariance matrix of the corresponding categories. Experiments with two datasets, Trento and Qingdao, resulted in ITS recall of 94% and 96%, accuracy of 82% and 91%, and F-scores of 0.87 and 0.93. Compared with the MS algorithm, our method is more accurate and less likely to under-segment individual trees in many cases. It can provide data support for the management and conservation of high-density multistoried mixed forest areas.