Heimdall: An Agentic Eye for Chemistry

State of The Art (SOTA) Optical Chemical Structure Recognition (OCSR) systems include graph-based recognizers, SMILES-generating models, chemistry Vision-Language Models (VLMs), and general multimodal Large Language Models (LLMs). When these systems encounter degraded scans, hand drawings, varied depiction styles, complex stereochemistry, organometallics, or Markush structures, they can hallucinate atoms, bonds, or entire molecules without reporting uncertainty or the assumptions behind the answer. We introduce Heimdall, a tool-using multi-agent system for chemical structure recognition and extraction. Reader agents zoom into uncertain regions and report every underlying assumption. A deterministic RDKit backend rejects chemically invalid graph assemblies. Heimdall defers an image when the available evidence cannot support an answer, and every such decision is backed by an auditable reason. Its multimodal patent pipeline combines text, figures, and page layout to distinguish claimed compounds, disclosed examples, and Markush definitions. This work is one of the largest OCSR studies, comparing Heimdall with 16 specialist OCSR systems and multimodal LLMs across a large and diverse benchmark set covering conventional and frontier chemistry.