To effectively maintain and analyze a large amount of real-time sensor data, one often uses a filtering technique that reflects characteristics of original data well. This paper proposes a novel method for recommending the measurement noise for Kalman filtering, which is one of the most representative filtering techniques. Kalman filtering corrects inaccurate values of input sensor data, and its filtering performance varies depending on the input noise parameters. In particular, if the noise parameters determined based on the user’s experience are incorrect, the accuracy of Kalman filtering may be reduced significantly. Based on this observation, this paper addresses how to determine the measurement noise variance, a major input parameter of Kalman filtering, by analyzing past sensor data and how to use the estimated noise to improve the filtering accuracy. More specifically, to estimate the measurement noise variance, two analytical methods are proposed: one a transform-based method using a wavelet transform and the other a learning-based method using a denoising autoencoder. Experimental results show that the proposed methods estimated the measurement noise variance accurately and were superior to the experience-based method in the filtering accuracy.

In this study, we proposed Smart Open Data as a Service (SODAS) as a new open data platform based on the international standards Data Catalog Vocabulary (DCAT) and Comprehensive Knowledge Archive Network (CKAN) to facilitate the release and sharing of data. We first analyze the five problems in the legacy CKAN and then draw up corresponding solutions through three core strategies: CKAN expansion, DCATv2 support, and extendable DataMap. We then define four components and nine function blocks of SODAS for each core strategy. As a result, SODAS drives Open Data Portal, Open Data Reference Model, DataMap Publisher, and Analytics and Development Environment (ADE) Provisioning for connecting the defined function blocks. We confirm that each function works correctly through the SODAS Web portal, and then we apply SODAS to actual data distribution sites to prove its efficiency and practical use. SODAS is the first open data platform that provides secure interoperability between heterogeneous platforms based on international standards, and it enables domain-free data management with flexible metadata.

Recently, there has been an explosive generation of streaming data in various fields such as IoT and network attack detection, medical data monitoring, and financial trend analysis. These domains require precise and rapid analysis capabilities by minimizing noise from continuously generated raw data. In this paper, we propose SPinDP (Stream Purifier in Distributed Platform), an open source-based high-speed stream purification platform, to support real-time stream purification. SPinDP consists of four major components, Data Stream Processing Engine, Purification Library, Plan Manager, and Shared Storage, and operates based on open-source systems including Apache Storm and Apache Kafka. In these components, stream processing throughput and latency are critical performance metrics, and SPinDP significantly enhances distributed processing performance by utilizing the ultra-high-speed network RDMA (Remote Direct Memory Access). For the performance evaluation, we use a distributed cluster environment consisting of nine nodes, and we show that SPinDP’s stream processing throughput is more than 28 times higher than that of the existing Ethernet environment. SPinDP also significantly reduces the processing latency by more than 2473 times on average. These results indicate that the proposed SPinDP is an excellent integrated platform that can efficiently purify high-speed and large-scale streams through RDMA-based distributed processing.

Each time-series has its own linear trend, the directionality of a timeseries, and removing the linear trend is crucial to get the more intuitive matching results. Supporting the linear detrending in subsequence matching is a challenging problem due to a huge number of possible subsequences. In this paper we define this problem the linear detrending subsequence matching and propose its efficient index-based solution. To this end, we first present a notion of LD-windows (LD means linear detrending), which is obtained as follows: we eliminate the linear trend from a subsequence rather than each window itself and obtain LD-windows by dividing the subsequence into windows. Using the LD-windows we then present a lower bounding theorem for the index-based matching solution and formally prove its correctness. Based on the lower bounding theorem, we next propose the index building and subsequence matching algorithms for linear detrending subsequence matching.We finally show the superiority of our index-based solution through extensive experiments.

Secure similar document detection (SSDD) identifies similar documents of two parties while each party does not disclose its own sensitive documents to another party. In this paper, we propose an efficient 2-step protocol that exploits a feature selection as the lower-dimensional transformation and presents discriminative feature selections to maximize the performance of the protocol. For this, we first analyze that the existing 1-step protocol causes serious computation and communication overhead for high dimensional document vectors. To alleviate the overhead, we next present the feature selection-based 2-step protocol and formally prove its correctness. The proposed 2-step protocol works as follows: (1) in the filtering step, it uses low dimensional vectors obtained by the feature selection to filter out non-similar documents; (2) in the post-processing step, it identifies similar documents only from the non-filtered documents by using the 1-step protocol. As the feature selection, we first consider the simplest one, random projection (RP), and propose its 2-step solution SSDD-RP. We then present two discriminative feature selections and their solutions: SSDD-LF (local frequency) which selects a few dimensions locally frequent in the current querying vector and SSDD-GF (global frequency) which selects ones globally frequent in the set of all document vectors. We finally propose a hybrid one, SSDD-HF (hybrid frequency), that takes advantage of both SSDD-LF and SSDD-GF. We empirically show that the proposed 2-step protocol outperforms the 1-step protocol by three or four orders of magnitude.