Generating Critic-Guided Synthetic Experience in Latent World Models

Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We introduce action refinement methodologies for experience generation in world models, including Critic-Guided Synthetic Experience (CGSE). Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the $Q$-function within a trust region. Then, our method uses a frozen latent world model to imagine the future state given each updated action and appends the one maximizing the $Q$-value under the critic as a synthetic transition to the replay buffer. Our method requires no additional environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of 7.1% on the average success rate.