Provocation: Who benefits from “inclusion” in Generative AI?

The demands for accurate and representative generative AI systems means there is an increased demand on participatory evaluation structures. While these participatory structures are paramount to to ensure non-dominant values, knowledge and material culture are also reflected in AI models and the subsequent generated content; we argue that dominant structures of community participation in AI development and evaluation are not explicit enough about the benefits and harms that members of socially marginalized groups may experience as a result of their participation. Without explicit interrogation of these benefits by AI developers, as a community we may remain blind to the immensity of systemic change that is needed as well. To support this provocation, we present a speculative case study, developed from our own collective experiences and based on personal positionalities. This is presented as a means to help itemize the barriers that need to be overcome as well in order for the proposed benefits to marginalized communities to be realized, and harms mitigated.