Hash tables are the fundamental data structure for analytical database workloads, such as aggregation, joining, set filtering and records deduplication. The performance aspects of hash tables differ drastically with respect to what kind of data are being processed or how many inserts, lookups and deletes are constructed. In this paper, we address some common use cases of hash tables: aggregating and joining over arbitrary string data. We designed a new hash table, SAHA, which is tightly integrated with modern analytical databases and optimized for string data with the following advantages: (1) it inlines short strings and saves hash values for long strings only; (2) it uses special memory loading techniques to do quick dispatching and hashing computations; and (3) it utilizes vectorized processing to batch hashing operations. Our evaluation results reveal that SAHA outperforms state-of-the-art hash tables by one to five times in analytical workloads, including Google’s SwissTable and Facebook’s F14Table. It has been merged into the ClickHouse database and shows promising results in production.
17 September 2024 in Computational Materials Science
The development of effective and comprehensive methods for mapping and monitoring reservoirs is essential for the utilization of water resources and flood control. Remote sensing has the great advantages of broad spatial coverage and regular revisit to meet the demand of large-scale and long-term tasks of earth observation. Although there already exist some methods for coarse-grained identification of reservoirs at region-level in remote sensing images, it remains a challenge to recognize and localize reservoirs accurately with insufficiency of object details and samples annotated. This study focuses on the fine-grained identification and location of reservoirs with a two-stage CNN framework method, which is comprised of a coarse classification between aquatic and land areas of image patches and a fine detection of reservoirs in aquatic patches with precise geographical coordinates. Moreover, a NIR RCNN detection network is proposed to make use of the multi-spectral characteristics of Sentinel-2 images. To verify the effectiveness of our proposed method, we construct a reservoir and dam dataset of 36 Sentinel-2 images which are sampled in various provinces across China and annotated at the instance level by manual work. The experimental results in the test set show that the two-stage CNN method achieves an average recall of 80.83% nationwide, and the comparison between reservoirs recognized by the proposed model and those provided by the China Institute of Water Resources and Hydropower Research verifies that the model reaches a recall of about 90%. Both the indicator evaluation and visualization of identification results have shown the applicability of the proposed method to reservoir recognition in remote sensing images. Being the first attempt to make a fine-grained identification of reservoirs at the instance level, the two-stage CNN framework, which can automatically identify and localize reservoirs in remote sensing images precisely, shows the prospect to be a useful tool for large-scale and long-term reservoir monitoring.
Data imputation is an effective way to handle missing data, which is common in practical applications. In this study, we propose and test a novel data imputation process that achieve two important goals: (1) preserve the row-wise similarities among observations and column-wise contextual relationships among features in the feature matrix, and (2) tailor the imputation process to specific downstream label prediction task. The proposed imputation process uses Transformer network and graph structure learning to iteratively refine the contextual relationships among features and similarities among observations. Moreover, it uses a meta-learning framework to select features that are influential to the downstream prediction task of interest. We conduct experiments on real-world large data sets, and show that the proposed imputation process consistently improves imputation and label prediction performance over a variety of benchmark methods.