One of the most important tasks of any platform for big data processing is storing the data received. Different systems have different requirements for the storage formats of big data, which raises the problem of choosing the optimal data storage format to solve the current problem. This paper describes the five most popular formats for storing big data, presents an experimental evaluation of these formats and a methodology for choosing the format. The following data storage formats will be considered: avro, CSV, JSON, ORC, parquet. At the first stage, a comparative analysis of the main characteristics of the studied formats was carried out; at the second stage, an experimental evaluation of these formats was prepared and carried out. For the experiment, an experimental stand was deployed with tools for processing big data installed on it. The aim of the experiment was to find out characteristics of data storage formats, such as the volume and processing speed for different operations using the Apache Spark framework. In addition, within the study, an algorithm for choosing the optimal format from the presented alternatives was developed using tropical optimization methods. The result of the study is presented in the form of a technique for obtaining a vector of ratings of data storage formats for the Apache Hadoop system, based on an experimental assessment using Apache Spark.

Web-surveys are one of the most popular forms of primary data collection used for various researches. However, mass surveys involve some challenges. It is required to consider different platforms and browsers, as well as different data transfer rates using connections in different regions of the country. Ensuring guaranteed data delivery in these conditions should determine the right choice of technologies for implementing web-surveys. The paper describes the solution to transfer a questionnaire to the client side in the form of an archive. This technological solution ensures independence from the data transfer rate and the stability of the communication connection with significant survey filling time. The conducted survey benefited the service of education psychologists under the federal Ministry of Education. School psychologists consciously took part in the survey, realizing the importance of their opinion for organizing and improving their professional activities. The desire to answer open-ended questions in detail created a part of the answers in the dataset, where there were several sentences about different aspects of professional activity. An important challenge of the problem is the Russian language, for which there are not as many tools as for the languages more widespread in the world. The survey involved 20,443 school psychologists from all regions of the Russian Federation, both from urban and rural areas. The answers did not contain spam, runaround answers, and so on as evidenced by the average response time. For the surveys, an authoring development tool DigitalPsyTools.ru was used.

One of the most popular methods for building analytical platforms involves the use of the concept of data lakes. A data lake is a storage system in which the data are presented in their original format, making it difficult to conduct analytics or present aggregated data. To solve this issue, data marts are used, representing environments of stored data of highly specialized information, focused on the requests of employees of a certain department, the vector of an organization’s work. This article presents a study of big data storage formats in the Apache Hadoop platform when used to build data marts.

One of the most important tasks of any platform for big data processing is the task of the storing data received. Different systems have different requirements for the storage formats of big data, which raises the problem of choosing the optimal data storage format to solve the current problem. This paper describes the five most popular formats for storing big data, presents an experimental evaluation of these formats and a methodology for choosing the format.

При разработке озера данных на таких платформах, как Apache Hadoop, важным вопросом становится выбор формата хранения данных. Этот выбор должен опираться на ряд различных критериев, одним из которых являются временные затраты при запуске различных запросов к этим данным. Однако любая система обработки данных предполагает постоянный рост объема этих данных. В связи с этим возникает необходимость изучения эффективности форматов в динамике роста объема данных, хранящихся в системе. В данной статье предлагается методика оценки эффективности форматов хранения данных в озерах данных, построенных на платформе Apache Hadoop, в динамике роста объема данных. Предложен эксперимент, представляющий из себя ряд запусков запросов различной сложности к данным, хранящихся в форматах JSON, Apache Avro, ORC, Apache Parquet. Для запуска запросов использовался фреймворк Apache Spark. When developing a data lake on platforms such as Apache Hadoop, the choice of data storage format becomes an important issue. This choice should be based on a number of different criteria, one of which is the time it takes to run different queries on this data. However, any data processing system assumes a constant growth in the volume of this data. In this regard, it becomes necessary to study the effectiveness of formats in the dynamics of growth in the amount of data stored in the system. This article proposes a methodology for assessing the effectiveness of data storage formats in data lakes built on the Apache Hadoop platform in the dynamics of data growth. An experiment is proposed, which is a series of queries of varying complexity to data stored in JSON, Apache Avro, ORC, Apache Parquet formats. The Apache Spark framework was used to run queries.