The estimation of correlation matrices in high-dimensional data streams presents a fundamental conflict between computational efficiency and statistical robustness. Moment-based estimators, such as Pearson’s correlation, offer linear complexity but lack robustness. In contrast, high-breakdown methods like the minimum covariance determinant (MCD) are computationally prohibitive () for real-time applications. This paper introduces Fast Outlier-Robust Correlation Estimation (FORCE), a streaming algorithm that performs adaptive coordinate-wise trimming using the algorithm for streaming quantile approximation, requiring only memory independent of stream length. We evaluate FORCE against six baseline algorithms—including exact trimmed methods (TP-Exact, TP-TER) that use sorting with storage—across five benchmark datasets spanning synthetic, financial, medical, and genomic domains. FORCE achieves speedups of approximately 470× over FastMCD and 3.9× over Spearman’s rank correlation. On S&P 500 financial data, coordinate-wise trimmed methods substantially outperform FastMCD: TP-Exact achieves the best RMSE (0.0902), followed by TP-TER (0.0909) and FORCE (0.1186), compared to FastMCD’s 0.1606. This result demonstrates that coordinate-wise trimming better accommodates volatility clustering in financial time series than multivariate outlier exclusion. FORCE achieves 76% of TP-Exact’s accuracy while requiring less memory, enabling robust estimation in true streaming environments where data cannot be retained for batch processing. We validate the 25% breakdown point shared by all IQR-based trimmed methods using the ODDS-satellite benchmark (31.7% contamination), confirming identical degradation for FORCE, TP-Exact, and TP-TER. For memory-constrained streaming applications with contamination below 25%, FORCE provides the only viable path to robust correlation estimation with bounded memory.
15 September 2017 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
During the development or deployment of discrete event simulation systems, many sudden changes in requirements emerge. To embrace the changes rapidly and reduce development costs, attaching reusable, and black-boxed components to existing systems has been regarded as one of the most effective approaches because modifications at the code level are generally costly and risky. However, since this approach requires components that are an exact fit, it may not be easy to avoid the modifications at the source code level actually. Moreover, the required components may not exist. Hence, this paper applies black-box extensibility to simulation systems in order to avoid the modifications. This paper proposes adaptive discrete event simulation systems using event control models for extending and modifying semantics through event-based simulation interface. With the proposed work, the simulation events are modulated, deleted, and generated by the event-oriented control functions in the event control model to embrace the changes of requirements. It can substitute for the modifications at the code level and extend the existing behavioral semantics. As a result, the proposed work provides a new alternative step in the development process with reusable components to avoid modifications at the source code level. The new step will lead to rapid adaptations of existing simulation systems. To support the effectiveness of this approach, this paper will describe applicable examples based on our empirical studies.
Although discrete-event simulation has been widely used in various engineering fields, its ef-ficiency remains an issue. Ranking and selection (R&S) procedures can solve this efficiency problem by allocating a limited simulation budget intelligently. While the existing R&S procedures mostly aim to find only the best simulation input design, practitioners sometimes require the worst design as well to analyze systems requiring high reliability, such as military systems, municipal waste management, etc. Motivated by these practical needs, we propose a simulation budget allocation procedure for selecting both extreme designs simultaneously in the presence of large stochastic noise. To maximize the accuracy of the selections under a limited budget, the proposed procedure sequentially allocates a small budget and updates the simulation re-sults such that they can be used as significant evidence for the correct selections. Our experimental results on benchmark and practical problems demonstrate improved efficiency compared to previous works. It is ex-pected that the proposed procedure will be effectively utilized in the fields of the fourth industrial revolution, such as digital twins that demand quickly finding both extreme designs to maintain synchronization with the corresponding real systems.
Modern society consists of various groups according to their respective interests. The importance of the citizen participation decision-making process in which such various groups get involved in the numerous decision-making of the society has been emerging. The living lab (LL) can be a sustainable approach in such a modern society because all stakeholders can participate in the problem-solving process. In LL, every group communicates, defines their problems, and discusses with experts to find the best solution. For this process to work effectively, the discussions should be based on clear scientific evidence instead of vague words. This study introduces the modeling and simulation (M&S) process to establish a theoretical basis to help the participants in LL identify problematic situations and analyze the solutions. This process involves discrete event system formalism with a set-theoretical modular form among various modeling and simulation theories and the execution environments. Based on them, participants can reuse or extend the existing simulation model to accelerate the problem-solving process of LL. The case study for multi-modal transit station analysis demonstrates the effectiveness of M&S in LL.