Evaluating LLMs for Automated SEM Powder Characterization Workflows

Powder morphology is critical to powder-based additive manufacturing, where particle size and shape influence deposition behavior and final part quality. While computer vision methods can automate SEM particle segmentation and morphology extraction, tailoring downstream analyses to specific characterization goals still frequently requires users to select or modify specialized analysis tools. To address this limitation, we evaluate LLM-guided automated characterization, where large language models (LLMs) interpret natural-language characterization requests using three strategies: raw-data inference, precomputed-summary retrieval, and deterministic-tool routing. Across 13 API-accessible and locally hosted LLM configurations and 93,379 characterization tasks, models achieved 92.0% accuracy when retrieving measurements from precomputed summaries, compared with 48.8% when computing them from raw particle records. For requests requiring additional analysis, schema-guided routing allowed models to select deterministic tools and arguments rather than compute directly, with 10 of 13 configurations scoring at least 95.0% routing accuracy. However, performance declined as tasks became more open-ended, indicating that LLM-guided characterization is most reliable for bounded analysis requests while open-ended morphology selection remains challenging.