Efficient bias mitigation in T2I diffusion models using Concept Graphs

Text-to-Image diffusion models often propagate harmful bias inherited from the training data. Existing bias mitigation techniques either intervene only at the text encoder or provide inference-time guidance, often leading to generations that collapse into semantically incoherent outputs. To address these limitations, we introduce CO-ALIGN (Concept Ontology Alignment), a novel bias mitigation approach based on concept-graph alignment which operates on the model's internal concept ontology. By aligning concepts within the text encoder and denoiser, CO-ALIGN achieves significant bias reduction while preserving generative integrity. We demonstrate the effectiveness of concept graph alignment in three paradigms- text-encoders, denoisers, and joint text-denoiser ontology alignment. CO-ALIGN outperforms outperforms the state of the art, improving fairness by 30\%, $\Delta FID=11.4$ in image quality, 2.8\% in image fidelity, all while reducing semantically incoherent outputs by 88\%. Beyond bias mitigation, we show that CO-ALIGN benefits other downstream tasks as well. In particular, our experiments demonstrate that better-aligned internal ontologies enhance concept unlearning robustness across multiple unlearning techniques.