Federated learning (FL) is a distributed machine learning framework that allows resource-constrained clients to train a global model jointly without compromising data privacy. Although FL is widely adopted, high degrees of systems and statistical heterogeneity are still two main challenges, which leads to potential divergence and nonconvergence. Clustered FL handles the problem of statistical heterogeneity straightly by discovering the geometric structure of clients with various data generation distributions and getting multiple global models. The number of clusters contains prior knowledge about the clustering structure and has a significant impact on the performance of clustered FL methods. Existing clustered FL methods are inadequate for adaptively inferring the optimal number of clusters in environments with high systems’ heterogeneity. To address this issue, we propose an iterative clustered FL (ICFL) framework in which the server dynamically discovers the clustering structure by successively performing incremental clustering and clustering in one iteration. We focus on the average connectivity within each cluster and give incremental clustering and clustering methods that are compatible with ICFL based on mathematical analysis. We evaluate ICFL in experiments on high degrees of systems and statistical heterogeneity, multiple datasets, and convex and nonconvex objectives. Experimental results verify our theoretical analysis and show that ICFL outperforms several clustered FL baseline methods.