Leveraging Large Language Models for Explaining Material Synthesis Mechanisms: The Foundation of Materials Discovery

Large language models (LLMs) have attracted significant attention in the advancement of materials discovery, particularly in their role in automation and robotics. However, a key question remains: Do these models operate based on a true grasping of physicochemical principles when designing experiments or interpreting results? Existing evaluations primarily focus on fact-checking tasks such as material property prediction and named entity recognition, while neglecting the reasoning required to grasp fundamental synthesis mechanisms. Furthermore, no previous studies have directly evaluated LLMs’ ability to reason about synthesis mechanisms. To address these challenges, we first develop a benchmark containing 775 semi-manually created multiple-choice questions in the field of gold nanoparticles (AuNPs) synthesis for evaluation. Second, we probe the model's output logits to derive precise selection probabilities for the correct answers, obtaining a confidence-based score (c-score) as a quantitative evaluation metric. Additionally, based on this evaluation, we also develop an AI assistant using retrieval-augmented generation (RAG) to explain AuNP synthesis mechanisms, achieving a 10\% improvement in accuracy over the leading model, Claude. Our study highlights the potential of LLMs in recogniz scientific mechanisms and offers a valuable tool for aiding the exploration of more synthesis methods. Moreover, our dataset establishes a foundation for developing highly efficient models with the utilization of material synthesis mechanisms. Code and dataset are available \href{https://github.com/Dandelionym/llmformechanisms}{here}.