Decreasing Inconsistencies in Differentially Private Language Models through Self-Distillation

Differentially private SGD (DPSGD) enables training large language models (LLMs) on private data with a worst-case guarantee on data leakage, but this guarantee comes at the cost of model utility due to added noise\niloofar{we should add a sentence here quantifying the errors and explaining that there would be added inconsistencies as well}. We propose a novel approach that leverages DPSGD and knowledge distillation together to produce differentially private LLMs that have higher utility and produce more fluent, consistent text. Our experimental results show our method reduces hallucinations by an average of 90.79\%, and increases coherence and accuracy of generated text\niloofar{we need to explain the task here, its paraphrasing, and summarization. We should also add a sentence describing an intuition of why this works.