How Large Language Models Learn Materials Science

Large language models are increasingly applied to materials science, yet fundamental questions remain about their reliability and knowledge encoding. Evaluating 28 LLMs across four materials science tasks—over 400 base, fine-tuned and cross-task configurations—we find that output modality fundamentally determines model behavior. For symbolic tasks, fine-tuning converges to consistent, verifiable answers with reduced response entropy, while for numerical tasks, fine-tuning improves prediction accuracy but models remain inconsistent across repeated inference runs, limiting their reliability as quantitative predictors. Knowledge graph completion analysis reveals that fine-tuning teaches models the statistical answer landscape of the task rather than individual facts. For numerical regression, we find that better performance can be obtained by extracting embeddings directly from intermediate transformer layers than from model text output, revealing an "LLM head bottleneck," though this effect is property- and dataset-dependent. Finally, we present the first longitudinal study of GPT model performance in materials science, tracking four models over 18 months and observing 9–43% performance variation that poses reproducibility challenges for scientific applications.