The growth of data publishing, sharing, and mining mechanisms in various fields of industry and science has led to an increase in the flow of data, making it an important asset that needs to be protected and managed effectively. To this end, different mechanisms have been used across different domains, including Privacy Enhancing Technologies like Synthetic Data Generation, which aim to protect user-sensitive data and prevent misuse among different domains. Then, Synthetic data has been used not only to augment datasets and balance classes but also in applications of data analysis paradigms that aim to provide useful insights in terms of utility while preserving the privacy of sensitive data. Still, there is a gap in the conceptual and state-of-the-art understanding of the level of privacy synthetic data generators can provide and how they affect various industries and fields. This systematic review attempts to address how privacy has been assessed and measured in the framework of synthetic data generation, and getting to know which metrics have been used to evaluate those mechanisms. We provide an overview a total of 105 recent studies in this field after a screening process and identify future open research directions. The main findings include a high prevalence of differential privacy as a privacy-preserving technique and privacy budget cost as a trade-off metric, with a high percentage of GAN-based model implementations, and mainly healthcare applications. Our systematic review covers multiple privacy domains and can be understood as a general framework for privacy measurement applied in Synthetic Data Generation.

Synthetic data is becoming the way forward to manage legal and regulatory aspects of biomedical research involving personal and clinical data. As no matches are expected between artificial instances and real samples and/or subjects, external researchers performing secondary analyses could benefit significantly by having unlimited access to uncompromised information. In this context, one of the main objectives of the H2020 VITALISE project is to develop a platform for providing synthetic data generated from real data collected in Living Labs to those external researchers. In addition, while some time series specific synthetic data generation models exist, only a few of them consider metadata (e.g., patient demographics) as part of the time series generation process itself. Therefore, the objective of this research is to perform a comparative assessment of two synthetic data generation models that use and process the metadata of subjects differently: The Wasserstein GAN with Gradient Penalty (WGAN-GP) and the DöppelGANger (DGAN). To achieve this goal making sure the analyses were data-independent, we selected two healthcare-related longitudinal datasets: (1) Treadmill Maximal Effort Test (TMET) measurements from the University of Málaga; and (2) a hypotension subset derived from the MIMIC-III v1.4 database. After synthetic data was generated, we assessed three pivotal aspects: resemblance to the original data, utility, and level of privacy. As a main conclusion, the importance of using metadata as context variables and the methodology to take them into account was proved to be significant and valuable, the DGAN model offering better results overall. A more extensive time series specific evaluation is left as the main avenue for future research.

BACKGROUND Data scarcity and dispersion pose significant obstacles in biomedical research, particularly when addressing rare diseases. In such scenarios, synthetic data generation (SDG) has emerged as a promising path to mitigate the first issue. Concurrently, federated learning is a machine learning paradigm where multiple nodes collaborate to create a centralized model with knowledge that is distilled from the data in different nodes, but without the need for sharing it. This research explores the combination of SDG and federated learning technologies in the context of acute myeloid leukemia, a rare hematological disorder, evaluating their combined impact and the quality of the generated artificial datasets. OBJECTIVE This study aims to evaluate the privacy- and fidelity-related impact of horizontally federating SDG models in different data distribution scenarios and with different numbers of nodes, comparing them with centralized baseline SDG models. METHODS Two state-of-the-art generative models, conditional tabular generative adversarial network and FedTabDiff, were trained considering four different scenarios: (1) a nonfederated baseline with all the data available, (2) a federated scenario where the data were evenly distributed among different nodes, (3) a federated scenario where the data were unevenly and randomly distributed (imbalanced data), and (4) a federated scenario with nonindependent and identically distributed data distributions. For each of the federated scenarios, a fixed set of node quantities (3, 5, 7, 10) was considered to assess its impact, and the generated data were evaluated, attending to a fidelity-privacy trade-off. RESULTS The computed fidelity metrics exhibited statistically significant deteriorations (P<.001) up to 21% in the conditional tabular generative adversarial network and up to 62% in the FedTabDiff model due to the federation process. When comparing federated experiments trained with diverse numbers of nodes, no strong tendencies were observed, even if specific comparisons resulted in significative differences. Privacy metrics were mainly maintained while obtaining maximum improvements of 55% and maximum deteriorations of 26% between both models, although they were not statistically significant. CONCLUSIONS Within the scope of the use case scenario in this paper, the act of horizontally federating SDG algorithms results in a loss of data fidelity compared to the nonfederated baseline while maintaining privacy levels. However, this deterioration does not significantly increase as the number of nodes used to train the models grows, even though significative differences were found in specific comparisons. The different data partition distribution configurations had no significant effect on the metrics, as similar tendencies were found for all scenarios. CLINICALTRIAL

Background: Data scarcity and dispersion pose significant obstacles in biomedical research, particularly when addressing rare diseases. In such scenarios, synthetic data generation (SDG) has emerged as a promising path to mitigate the first issue. Concurrently, federated learning is a machine learning paradigm where multiple nodes collaborate to create a centralized model with knowledge that is distilled from the data in different nodes, but without the need for sharing it. This research explores the combination of SDG and federated learning technologies in the context of acute myeloid leukemia, a rare hematological disorder, evaluating their combined impact and the quality of the generated artificial datasets. Objective: This study aims to evaluate the privacy- and fidelity-related impact of horizontally federating SDG models in different data distribution scenarios and with different numbers of nodes, comparing them with centralized baseline SDG models. Methods: Two state-of-the-art generative models, conditional tabular generative adversarial network and FedTabDiff, were trained considering four different scenarios: (1) a nonfederated baseline with all the data available, (2) a federated scenario where the data were evenly distributed among different nodes, (3) a federated scenario where the data were unevenly and randomly distributed (imbalanced data), and (4) a federated scenario with nonindependent and identically distributed data distributions. For each of the federated scenarios, a fixed set of node quantities (3, 5, 7, 10) was considered to assess its impact, and the generated data were evaluated, attending to a fidelity-privacy trade-off. Results: The computed fidelity metrics exhibited statistically significant deteriorations (P<.001) up to 21% in the conditional tabular generative adversarial network and up to 62% in the FedTabDiff model due to the federation process. When comparing federated experiments trained with diverse numbers of nodes, no strong tendencies were observed, even if specific comparisons resulted in significative differences. Privacy metrics were mainly maintained while obtaining maximum improvements of 55% and maximum deteriorations of 26% between both models, although they were not statistically significant. Conclusions: Within the scope of the use case scenario in this paper, the act of horizontally federating SDG algorithms results in a loss of data fidelity compared to the nonfederated baseline while maintaining privacy levels. However, this deterioration does not significantly increase as the number of nodes used to train the models grows, even though significative differences were found in specific comparisons. The different data partition distribution configurations had no significant effect on the metrics, as similar tendencies were found for all scenarios.