Beyond Prediction Error: A Wind Turbine Case Study for CFD Surrogate Model Evaluation

Computational fluid dynamics (CFD) surrogate models are commonly evaluated using errors in predicted flow fields or aerodynamic coefficients. However, these intermediate-output errors do not necessarily indicate whether a surrogate can support the engineering decision for which it is intended. This paper demonstrates this distinction using the OB-GNN surrogate model in a wind-turbine case study. Errors in the predicted lift and drag coefficients are propagated through blade element momentum theory to the downstream decision metric, the power coefficient $C_P$. The analysis compares a broad dataset containing a variety of airfoils with a narrow dataset containing small geometric deviations of a NACA 0012 airfoil at a fixed operating condition. The $C_P$ prediction performance is nearly identical for the two datasets. However, the deviated airfoils produce much smaller differences in $C_P$, making them more difficult for the surrogate to distinguish. Using a 95% probability criterion, the surrogate reliably identifies the higher-performing airfoil in 89% of pairs from the broad dataset, compared with only 7% from the deviated-airfoil dataset. These results show that surrogate accuracy must be evaluated relative to the differences relevant to the intended engineering decision.