This paper aims to test the hypothesis that the quality of social media bot detection systems based on supervised machine learning may not be as accurate as researchers claim, given that bots have become increasingly sophisticated, making it difficult for human annotators to detect them better than random selection. As a result, obtaining a ground-truth dataset with human annotation is not possible, which leads to supervised machine-learning models inheriting annotation errors. To test this hypothesis, we conducted an experiment where humans were tasked with recognizing malicious bots on the VKontakte social network. We then compared the “human” answers with the “ground-truth” bot labels (‘a bot’/‘not a bot’). Based on the experiment, we evaluated the bot detection efficiency of annotators in three scenarios typical for cybersecurity but differing in their detection difficulty as follows: (1) detection among random accounts, (2) detection among accounts of a social network ‘community’, and (3) detection among verified accounts. The study showed that humans could only detect simple bots in all three scenarios but could not detect more sophisticated ones (p-value = 0.05). The study also evaluates the limits of hypothetical and existing bot detection systems that leverage non-expert-labelled datasets as follows: the balanced accuracy of such systems can drop to 0.5 and lower, depending on bot complexity and detection scenario. The paper also describes the experiment design, collected datasets, statistical evaluation, and machine learning accuracy measures applied to support the results. In the discussion, we raise the question of using human labelling in bot detection systems and its potential cybersecurity issues. We also provide open access to the datasets used, experiment results, and software code for evaluating statistical and machine learning accuracy metrics used in this paper on GitHub.

This paper presents behavioral analysis of resource allocation systems in cloud infrastructure. This analysis is performed on the basis of these systems monitoring and machine learning. To do it, the system creates a model of normal system behavior using learning set. The constructed model is used as a reference. Recognition of an abnormal behavior of the system is based on the comparing of the normal behavior with the current one. The results represent the beginning of the research. In the continuation, the issues of anomaly detection based on virtual machine (instance) resource utilization metrics will be analyzed. Also, an algorithm for determining anomalies will be developed based on a comprehensive analysis of the CPU, RAM, HDD resources usage. For enhancing of the anomalies detection accuracy it is planned to increase the period of learning.