9 January 2022 in International Journal of Intelligent Systems
In crowdsourcing scenarios, we can often obtain each instance's multiple noisy labels from different crowd workers and then use a label integration method to infer its integrated label. In spite of the effectiveness of label integration methods, a certain level of label noise still exists in integrated labels. To reduce the impact of label noise, noise correction has attracted much attention from researchers, and therefore a certain number of noise correction methods were proposed in recent years. Among them, Between-class Margin-based Noise Correction (BMNC) has demonstrated remarkable denoising performance. However, BMNC can only handle binary classification tasks. For multiclass classification tasks, we propose an effective but very simple noise correction method in this paper. We refer to our proposed method as Label Distribution-based Noise Correction (LDNC). At first, LDNC transforms each instance's multiple noisy labels into a label distribution. Then, LDNC uses the margin between the first- and second-largest label probabilities in the label distribution to identify and filter each possible noise instance and thus obtains a clean set and a noise set. Finally, LDNC builds a classifier on the clean set to relabel all instances in the noise set. The experimental results on 16 simulated and one real-world multiclass crowdsourced data sets show that LDNC significantly outperforms all the other existing state-of-the-art noise correction methods.
22 May 2024 in IEEE Transactions on Services Computing
Crowdsourcing services provide a fast and cheap way to annotate instances by employing crowd workers on the Internet. As a result, many learning from crowds (LFC) methods have been proposed in recent years. However, almost all these methods assume that all instances are benign, which makes them vulnerable to adversarial attacks. To improve the model's robustness to adversarial attacks, adversarial LFC (A-LFC) has attracted remarkable attention. A-LFC iteratively updates the true labels' estimations and the trained model by adversarial learning and uses the model's predictions in turn to help estimate the true labels. In A-LFC, to the best of our knowledge, the true labels' estimations and the model's predictions are inaccurate and the stopping condition for iterations is rough, which limit the model's performance. To further improve A-LFC, this paper proposes weighted A-LFC (WA-LFC). To reduce the impact of misinformation in the true labels' estimations and model's predictions, our method weights instances for adversarial learning and weights the model's predictions for estimating the true labels. Our method iteratively updates these weights and uses instance-weighted cross-entropy loss to decide when the iterative process should be stopped. Experiments on three real-world datasets show that our method substantially improves the trained model's performance. On average, the test accuracy of our method outperforms that of A-LFC by 10.06% and 27.55% in white-box and black-box attack settings, respectively.