Generating and Validating Agent and Environment Code for Simulating Realistic Personality Profiles with Large Language Models

Scaling up the creation of realistic agents and environments poses a significant challenge in artificial intelligence and science more broadly. Environments are traditionally handcrafted, restricting their scalability and diversity. Recent advancements in large language models (LLMs) offer a promising approach to automating environment design. In behavioral and cognitive science, a key goal is to characterize traits that predict behavior, typically through carefully designed cognitive tasks. However, this approach also faces significant scaling challenges due to the extensive human validation required. To address these limitations, we leverage LLMs to generate not only the code for environmental affordances and the agent policy but also the code that ensures their validity. Specifically, we assign agents distinct personality profiles based on data from large-scale psychological studies that have identified consistent and reliable personality traits. The LLM-generated validation code then evaluates how accurately these traits are reflected in the agents' simulated behaviors using the widely recognized HEXACO personality inventory. Our results demonstrate that the LLM-generated pipeline can simulate a diverse range of personality profiles. Additionally, we find that specific components, such as the type of contextual information in the LLM prompts, significantly impair the recoverability of these personality profiles. We believe our approach offers a systematic and scalable method for simulating realistic personality profiles by validating environments and agents generated by LLMs.