Data chunking algorithms divide data into several small data chunks in a certain way, thus transforming the operation of data into the one of multiple small data chunks. Data chunking algorithms have been widely used in duplicate data detection, parallel computing and other fields, but it is seldom used in data incremental synchronization. Aiming at the characteristics of incremental data synchronization, this paper proposes a novel data chunking algorithm. By dividing two data that need synchronization into small data chunks, comparing the contents of these small data chunks, different ones are the incremental data that need to be found. The new algorithm determines to set a cut-point based on the number of 1 contained in the binary format of all bytes in an interval. Thus it improves the resistance against the byte shifting problem at the expense of the chunk size stability, which makes it more suitable for the incremental data synchronization. Comparing this algorithm with several known classical or state of art algorithms, experiments show that the incremental data found by this algorithm can be reduced by 32%~57% compared to the others with same changes between two data. The experimental results based on real-world datasets show that PCI improves the calculation speed of classic Rsync algorithm up to 70%, however, with a drawback of increasing the Transmission compression rate up to 11.8%.
22 July 2021 in Concurrency and Computation: Practice and Experience
Bloom filter is a useful data structure, which is often used in the membership query with allowing errors. However, high computational cost of the hash functions limits the performance of the Bloom filter. In this paper, we propose a new Bloom filter based on a single hash function named No-partition Single-hashing Bloom filter (NPSHBF). Compared with the Standard Bloom filter (SBF), we theoretically prove that the false positive probability of NPSHBF is approximately equal to SBF. At the same time, we theoretically prove that the processes of modulo are independent to each other, which greatly improves the querying performance of the Bloom filter. After theoretical verification, we can see from a series of experimental results that the false positive probability of NPSHBF is consistent with the theoretical speculation, and the querying efficiency and generating efficiency of NPSHBF are much higher than SBF.
1 November 2020 in Journal of Physics: Conference Series
Safety production management system is widely used at present. The current safety production management system only improves some aspects of production safety, but it is not suitable for all engineering management, and there is no good solution in the engineering management mode. In this paper, a four-stage management mode based on safety production affairs is proposed, which plays a good role in the engineering management of safety production. The intelligent safety assessment system has a good effect.