On the cognitive alignment between humans and machines

In this paper, we explore the psychological relevance, similarity to brain representations, and subject-invariance of latent space representations in generative models. Using fMRI data from four subjects who viewed over 9,000 visual stimuli, we conducted three experiments to investigate this alignment. First, we assessed whether a linear mapping between the latent space of a VDVAE model and fMRI brain responses could accurately capture cognitive properties, specifically emotional valence, of the visual stimuli presented to both humans and machines. Second, we examined whether perturbing psychologically relevant dimensions in either the generative model or human brain data would produce corresponding cognitive effects in both systems—across models and human subjects. Third, we investigated whether a nonlinear mapping, approximated via a Taylor expansion up to the fifth degree, would outperform linear mapping in aligning cognitive properties. Our findings revealed three key insights: (1) the latent space of the generative model, in this case a very deep VAE, aligns with fMRI brain responses across all subjects tested (r ~ 0.4), (2) perturbations in the psychologically relevant dimensions of both the fMRI data and the generative model resulted in highly consistent effects across the aligned systems (both the model and human subjects), and (3) a linear mapping, approximated using Ridge regression, performed as well as or better than all Taylor expansions we tested. Together, these results suggest a universal cognitive alignment between humans and between human-model systems. This universality holds significant potential for advancing our understanding of basic cognitive processes and offers promising new avenues for studying mental disorders.