Characterizing Context Memorization and Hallucination of Language Models

Although Large Language Models (LLMs) have achieved remarkable performance in numerous downstream tasks, their proliferation has raised two significant concerns. One is that LLMs can hallucinate by generating content that contradicts relevant contextual information; the other is that LLMs can inadvertently leak private information due to memorization. Many prior works have extensively studied each concern independently, but none have investigated them simultaneously. Furthermore, analyzing the memorization of provided context during open-ended generation is understudied. To this end, we comprehensively characterize the memorization and hallucination of contextual information during summarization. Our analysis suggests that amplifying the context (by factoring out prior knowledge) and the context being out of distribution with respect to prior knowledge results in larger memorization of the context. We corroborate our analytical findings with experimental evaluations that show improving the F1 ROGUE-L score on CNN-DM for LLaMA 3 by $\textbf{10}$% over regular decoding requires $\textbf{1.5x}$ more memorization. Moreover, we empirically evaluate how memorization and hallucination are affected by (1) model capacity, (2) context size, (3) the length of the current response, and (4) different n-grams of the context. Our results hope to inform practitioners of the interplay between context memorization and hallucination for trustworthy deployment of LLMs.