Profile-Conditioned Travel User Simulation: An LLM-Based Framework with Multi-Dimensional Realism Evaluation
Peitong Li ⋅ Aleksandr Umnov ⋅ Fengjun Wang
Abstract
User simulations are crucial for scaling offline evaluation of conversational systems and data generation, but they often mismatch real traffic distributions and produce detectably ``LLM-like'' language. We present a profile-conditioned user simulation framework for a real-life AI trip-planner chatbot (AITP) at a large-scale online travel platform. The method constructs comprehensive profile cards from conversation logs, encoding behavioral attributes and travel preferences to steer traveler intents and trip entity choices and maintain cross-turn consistency. To reduce stylistic artifacts, we introduce Real Pattern Enhancement, which distills human-vs-LLM style differences from real--synthetic contrastive pairs and injects the resulting patterns into the simulator context. The simulator generates multi-turn dialogues in a closed loop by interacting with AITP. We evaluate realism along three dimensions: trip entity distribution alignment (Jensen--Shannon divergence), profile--dialogue consistency, and perceptual authenticity (pointwise and pairwise judgments). On 1{,}000 users, profile conditioning improves distributional alignment (JS $0.2227$ to $0.0586$) and profile faithfulness, while Real Pattern Enhancement reduces detectability to near chance (AUC $\approx 0.48$; paired accuracy $\approx 51\%$). The combined system yields scalable, human-like, profile-faithful travel dialogues. We additionally conduct a blind human study with two annotators on 50 real--synthetic dialogue pairs per setting. Without RPE, annotators identify Profile-only pairs with 98--100\% accuracy; with Profile+RPE, accuracy falls to 50--62\%, corroborating improved perceptual realism.
Chat is not available.
Successful Page Load