Dialogue with the Machine and Dialogue with the Art World: A Method for Evaluating AI as a Tool for Creativity

This paper proposes a novel dialogic and experimental evaluation method for generative AI tools in the context of creativity. Expanding beyond traditional evaluations such as benchmarks, user studies with crowd-workers, or focus groups conducted with artists, we draw on sociologist Howard Becker's concept of \emph{Art Worlds} to demonstrate dialogue as a methodology for evaluation. We present two mutually informed dialogues: 1) dialogues with art worlds', placing artists in conversation with experts such as art historians, curators, archivists, and AI researchers, and 2)dialogues with the machine' facilitated through structured artist- and critic-led experimentation with state-of-the-art generative AI tools. We demonstrate the value of our method through a case study with artists and experts steeped in non-western art worlds, specifically the Persian Gulf. We trace how these dialogues help create culturally rich and situated forms of evaluation for representational possibilities of generative AI that mimic the reception of generative artwork in the broader art ecosystem. They also allow artists to shift their use of the tools to respond to their cultural and creative context. Our study can provide generative AI researchers an understanding of the complex dynamics of technology, human creativity and the socio-politics of art worlds, to build more inclusive machines for diverse art worlds.