Traits from Trajectories: Measuring LLM Agent Personality from Behavior

LLM personality research adapts psychometric instruments to describe models as human-like entities, promising both a better account of what interacting with an agent is like and a principled way to match tasks to models. Prior work, however, finds that a model's questionnaire self-reports diverge from its actual behavior, leaving us without a trait measure that generalizes across scenarios. We therefore characterize LLM personality from observable behavior rather than self-report. We collect 529,256 real-world agent trajectories spanning 44 models, 10 tasks, and 36 harnesses, and extract 11,540 candidate features covering both what a model does at each step (functional) and the language it uses while doing it (linguistic). We screen these features against three psychometric criteria in sequence: split-half reliability, cross-situational consistency, and between-model discriminability. 207 features survive, describing how a model plans, reflects, and repeats itself alongside linguistic dimensions such as politeness and hedging, and they yield a distinct profile per model. For instance, GPT-4.1 plans little and leans on politeness, while Gemini-2.5-Pro reflects after acting and hedges. Measuring agents by what they do rather than what they say about themselves offers a route to user-centric evaluation and to routing tasks toward the models whose profiles fit them.