Keeping the Agent Out of the Hot Loop: Self-Healing DFT Workflows Across HPC Centers

High-throughput Density Functional Theory (DFT) workflows are widely used in computational chemistry, materials science, and condensed matter physics, but large-scale runs still require substantial human intervention when calculations, workflow daemons, queues, or archival steps fail. LLM agents can help reduce this supervision, but our deployment experience shows that giving them direct control to launch, cancel, or modify jobs risks wasting expensive compute or human debugging time. Motivated by this, we introduce LeDFTBuddy, an agentic monitoring and build workflow that helps construct a deterministic self-healing control plane while keeping LLM agents outside the production hot path. LeDFTBuddy observes the workflow state through typed read-only tools, diagnoses unresolved incidents, writes issues and draft PRs, and routes proposed fixes through adversarial review and human approval before deployment. The multi-agent workflow helped build and harden a deterministic outer loop for common DFT workflow failures and provide a queue migration policy across computational infrastructure for stalled jobs. Across multiple hardware partitions and HPC centers, the outer loop observed 1,677 remote workflow errors, recovered 731 of them with deployed logic, backfilled 8,343 missing records, and ran 15 queue migrations without duplicate live executions. The main lesson is that agents are useful for finding missing policies, but state-changing recovery for large-scale scientific workflows should remain deterministic or human-approved.