MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad memory bias, where familiar content improves reasoning performance, and the Strong Parametric Shortcut Hypothesis, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce MemoReason, a human-curated benchmark that pairs factual reasoning tasks with structurally identical fictional versions where real entities like people, companies, or dates are systematically replaced by fictional ones of the same type. This keeps task complexity constant while varying the familiarity of the context, allowing controlled measurement of how parametric memory affects reasoning. Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7% in the fictional setting, demonstrating a clear memory bias. However, a targeted analysis of failed fictional questions shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. MemoReason provides a controlled framework for studying these mechanisms and for extending paired factual–fictional evaluation to broader reasoning settings.