The strong background noise largely interferes with the feature extraction results of early faults of rolling bearings and the number of decomposed modes in the feature mode decomposition (FMD) algorithm depends on empirical selection. An overlapping group sparsity (OGS) method has an unsatisfactory performance in noise reduction. Therefore, in order to solve the above problems, an adaptive feature mode decomposition (AFMD) method was proposed in this study. In AFMD, the number of decomposed modes was determined with the singular value distribution curve to establish the average fractal dimension determination criterion for the selection of sensitive modes. In addition, a nonconvex overlapping group sparsity (NCOGS) noise reduction method was proposed. In NCOGS, a nonconvex penalty function was introduced into the regularization term of the OGS method to enhance the sparsity of the signal, so that the regularization parameter and the convexity parameter in NCOGS could be screened with the standard deviation of noise and permutation entropy values, respectively. Then, AFMD and NCOGS were combined together to extract the features of early faults of rolling bearings. Simulation and experimental results showed that the combined method effectively suppressed the noise interference generated in feature mode decomposition and improved the feature extraction capability of early faults of rolling bearings.