Differential Privacy vs. Empirical Privacy: Benchmarking Tabular Synthetic Data Generators

Differentially private synthetic data generators (DP-SDGs) provide formal privacy guarantees, but existing evaluations typically take these guarantees at face value rather than measuring empirical protection. We benchmark statistical and neural SDGs, with and without DP, on the Adult Census dataset, evaluating fidelity, utility, and privacy via membership inference attacks (MIAs). Both DP generators reduce privacy leakage relative to their non-DP baseline, and an empirical audit of their formal privacy guarantee confirms both stay well within budget. However, the trade-off differs by family as the statistical DP-SDG incurs smaller fidelity and utility losses relative to its non-DP baseline than the neural DP-SDG for a similar privacy gain. Along the way, we observe DP-related anomalies in a widely used generation library, which we are working to report to its developers, and switch to a correctly calibrated alternative to ensure the integrity of our results.