Multimodal CAD Specifications: Systematic Simulation and Evaluation of Communication Strategies for Human-AI Design Alignment

Visual design programming is vital for the development of physical and digital products. Example programmatic representations include mechanical design (e.g, CAD) and user interface design (e.g., SwiftUI, HTML/CSS), allowing for precise control over complex visual specifications. While Large Language Model (LLM) agents have demonstrated utility in general programming tasks, they struggle with the unique challenges of visual design programming, specifically the significant gap in spatial reasoning. Bridging this inherent gap between humans and AI agents requires effective communication to align human design intent with AI agents. To enable evaluations of how different communication strategies influence intent alignment without relying on information that is difficult to obtain from real-world iterative design sequences, we introduce a scalable and controlled simulation framework. It can synthesize design revision sequences and evaluate various human-AI agents interaction behaviors, such as describing designs in different level of details, annotating key changes and verifying the design outcome. Our simulation and evaluation show that while AI agents provides baseline alignment, human-in-the-loop interaction strictly maximizes geometric accuracy, informing a co-design prototype where automated multimodal specification generation assists and strengthens human supervision.