Recommending out-of-vocabulary (OOV) items is a challenging problem since the in-vocabulary (IV) items have well-trained behavioral embeddings but the OOV items only have content features. Current OOV recommendation models often generate "makeshift" embeddings for OOV items from content features and then jointly recommend with the "makeshift" OOV item embeddings and the behavioral IV item embeddings. However, merely using the "makeshift" embedding will result in suboptimal recommendation performance due to the substantial gap between the content feature and the behavioral embeddings. To bridge the gap, we propose a novel User Sequence IMagination (USIM) fine-tuning framework, which can imagine the user sequences and then refine the generated OOV embeddings with user behavioral embeddings. Specifically, we frame the user sequence imagination as a reinforcement learning problem and develop a custom recommendation-focused reward function to evaluate to what extend a user can help recommend the OOV items. Besides, we propose embedding-driven transition function to model the embedding transition after imaging a user. USIM has been deployed on a prominent e-commerce platform for months, offering recommendations for millions of OOV items and billions of users. Extensive experiments demonstrate that USIM outperforms traditional generative models in terms of cold-start performance across collaborative filtering and GNN-based collaborative filtering models.