The non‐independent and identically distributed (non‐IID) nature of data across different sites and the dynamic participation of client nodes represent two critical challenges faced by distributed industrial visual defect detection systems. While asynchronous federated learning (AFL) reduces training inefficiencies, its asynchronous update process makes the global model more vulnerable to data heterogeneity. This research proposes a contrastive clustering–based asynchronous federated learning (CC‐AFL) framework. The method integrates contrastive learning into the client clustering process, working simultaneously across instance‐level and cluster‐level spaces to identify hidden similarity patterns among participating clients. The server maintains update queues for each cluster and then applies staleness‐weighted aggregation based on the combined timestamp. Meanwhile, the framework uses client scheduling based on cluster queue lengths, giving priority to clusters with fewer updates to balance update frequencies across different clusters. Without requiring raw data sharing, the framework achieves improvements in both convergence speed and stability of AFL under changing scenarios. On the furniture board dataset, CC‐AFL achieves better performance with an average area under the receiver operating characteristic curve (I‐AUROC) of 74.38%, showing a 2.22% improvement over AFL‐DCS under different non‐IID settings. Experimental results show that CC‐AFL maintains stable convergence under different conditions, confirming the effectiveness and practical value of contrastive clustering and scheduling in AFL.