Title: Differentially Private Learning with Margin Guarantees

Abstract:

Preserving privacy is a crucial objective for machine learning algorithms. But, despite the remarkable theoretical and algorithmic progress in differential privacy over the last decade or more, its application to learning still faces several obstacles.

A recent series of publications have shown that differentially private PAC learning of infinite hypothesis sets is not possible, even for common hypothesis sets such as that of linear functions. Another rich body of literature has studied differentially private empirical risk minimization in a constrained optimization setting and shown that the guarantees are necessarily dimension-dependent. In the unconstrained setting, dimension-independent bounds have been given, but they admit a dependency on the norm of a vector that can be extremely large, which makes them uninformative.

These results raise some fundamental questions about private learning with common high-dimensional problems: is differentially private learning with favorable (dimension-independent) guarantees possible for standard hypothesis sets?

This talk presents a series of new differentially private algorithms for learning linear classifiers, kernel classifiers, and neural-network classifiers with dimension-independent, confidence-margin guarantees.

Joint work with Raef Bassily and Ananda Theertha Suresh.

Synthetic data and privacy.

Papers:

1. Generating Synthetic Datasets by Interpolating along Generalized Geodesics
2. Synthetic Clinical Trial Data while Preserving Subject-Level Privacy
3. Stutter-TTS: Synthetic Generation of Diverse Stuttered Voice Profiles
4. ReSPack: A Large-Scale Rectilinear Steiner Tree Packing Data Generator and Benchmark
5. Visual Pre-training for Navigation: What Can We Learn from Noise?
6. HAPNEST: An efficient tool for generating large-scale genetics datasets from limited training data
7. Improving dermatology classifiers across populations using images generated by large diffusion models
8. Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding
9. Leading by example: Guiding knowledge transfer with adversarial data augmentation
10. Exploring Biases in Facial Expression Analysis
11. Noise-Aware Statistical Inference with Differentially Private Synthetic Data
12. A source data privacy framework for synthetic clinical trial data
13. Approaches to Optimizing Medical Treatment Policy using Temporal Causal Model-Based Simulation
14. Entity-Controlled Synthetic Text Generation using Contextual Question and Answering with Pre-trained Language Models
15. Distributional Privacy for Data Sharing
16. Vine Copula Based Data Generation for Machine Learning With an Application to Industrial Processes
17. Systematic review of effect of data augmentation using paraphrasing on Named entity recognition
18. PRISIM: Privacy Preserving Synthetic Data Simulator
19. Fair Synthetic Data Does not Necessarily Lead to Fair Models
20. FARE: Provably Fair Representation Learning

Papers:

1. SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data
2. MAQA: A Multimodal QA Benchmark for Negation
3. Generating Realistic Synthetic Relational Data through Graph Variational Autoencoders
4. Mitigating Health Data Poverty: Generative Approaches versus Resampling for Time-series Clinical Data
5. Private GANs, Revisited
6. Contrastive Learning on Synthetic Videos for GAN Latent Disentangling
7. HandsOff: Labeled Dataset Generation with No Additional Human Annotations
8. Secure Multiparty Computation for Synthetic Data Generation from Distributed Data
9. Counterfactual Fairness in Synthetic Data Generation
10. HyperTime: Implicit Neural Representations for Time Series
11. Generic and Privacy-free Synthetic Data Generation for Pretraining GANs
12. Importance of Synthesizing High-quality Data for Text-to-SQL Parsing
13. Fast Learning of Multidimensional Hawkes Processes via Frank-Wolfe
14. TAPAS: a Toolbox for Adversarial Privacy Auditing of Synthetic Data
15. On the legal nature of synthetic data
16. Synthesizing Informative Training Samples with GAN
17. Unsupervised Anomaly Detection for Auditing Data and Impact of Categorical Encodings.
18. C-GATS: Conditional Generation of Anomalous Time Series
19. Medical Scientific Table-to-Text Generation with Synthetic Data under Data Sparsity Constraint
20. Mind Your Step: Continuous Conditional GANs with Generator Regularization
21. Federated Learning on Patient Data for Privacy-Protecting Polycystic Ovary Syndrome Treatment
22. Random Walk based Conditional Generative Model for Temporal Networks with Attributes
23. Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization
24. Conditional Progressive Generative Adversarial Network for satellite image generation
25. Multi-Modal Conditional GAN: Data Synthesis in the Medical Domain
26. Hypothesis Testing using Causal and Causal Variational Generative Models

Compared to other machine learning tasks like classification and regression, synthetic data generation is a new area of inquiry for machine learning. One challenge we encountered early on in working with synthetic data was the lack of standardized metrics for evaluating it. Although evaluation for tabular synthetic data is less subjective than for ML-generated images or natural language, it comes with its own specific considerations. For instance, metrics must take into account what the data is being generated for, as well as tradeoffs between quality, privacy, and utility that are inherent to this type of data.

To begin addressing this need, we created an open source library called SDMetrics, which contains a number of synthetic data evaluation tools. We identified inherent hierarchies that exist in these evaluations — for example, columnwise comparison vs. correlation matrix comparison — and built ways to test and validate these metrics. The library also provides user-friendly, focused reports and mechanisms to prevent "metric fatigue.”

Synthetic data and fairness.

Progress, achievements and challenges in synthetic data.

Synthetic medical data – needed to bring ML in medicine up to speed?

To bring medical AI to the next level, in terms of exceeding state-of-art and not just developing but also implementing novel algorithms as decision support tools into clinical practice, we need to bring data scientists, laboratory scientists and physician (researchers) together. To succeed with this, we need to bring medical data out in the open, to benefit from the best possible models being developed jointly by the data science community. Solving the data privacy issues – in terms of creating synthetic data sets reflecting the correlations of the original data sets without jeopardizing data privacy – is needed. Here you get the perspectives on real clinical challenges, real data science approaches being implemented into clinical practice, regulatory issues and real unmet needs of synthetic data sets from the perspective of a physician researcher.

Title: The Fifth Paradigm of Scientific Discovery

Abstract: I will argue that we may be at the beginning of a new paradigm of scientific discovery based on deep learning combined with ab initio simulation of physical processes. We envision a system where simulations generate data to train neural surrogate models that in turn will accelerate simulations. The result will be an active learning framework where accurate data is acquired when the surrogate model is uncertain about it’s predictions. We will argue this hybrid approach can accelerate scientific discovery, for instance the the search for new drugs, and materials.

Progress, achievements, and challenges in synthetic data.

Privacy-Preserving Data Synthesis for General Purposes

The recent success of deep neural networks (DNNs) hinges on the availability of large-scale datasets; however, training on such datasets often poses privacy risks for sensitive training information, such as face images and medical records of individuals. In this talk, I will mainly discuss how to explore the power of generative models and gradient sparsity, and talk about different scalable privacy-preserving generative models in both centralized and decentralized settings. In particular, I will introduce our recent work on large-scale privacy-preserving data generative models leveraging gradient compression with convergence guarantees. I will also introduce how to train generative models with privacy guarantees in heterogeneous environments, where data of local agents come from diverse distributions. We will finally discuss some potential applications for different privacy-preserving data synthesis strategies.