PrefDisco: Evaluating Proactive Personalization through Interactive Preference Discovery

Current language models struggle to discover user preferences through conversation, often producing responses that mismatch individual needs. We introduce PrefDisco, a meta-benchmark framework that transforms existing benchmarks into interactive personalization tasks using psychologically-grounded personas with consistent preference patterns. Evaluation of 22 frontier models across ten tasks reveals systematic failures. Counterintuitively, 42.6% of model-task combinations perform worse when attempting personalization than providing generic responses. We show that models tend not to ask questions even when provided the option to, even though question asking improves preference alignment. Domain analysis reveals optimization brittleness: mathematical reasoning suffers severe degradation under personalization (3.5% accuracy loss), while social reasoning maintains robustness (3.1% gain). These findings establish interactive preference discovery as a distinct capability requiring dedicated architectural innovations rather than an emergent property of general language understanding.